Spaces:
Running
on
CPU Upgrade
Running
on
CPU Upgrade
Commit
·
89f19e4
1
Parent(s):
da48ebe
Initial commit: n8n infrastructure with AI and automation
Browse filesAdd production-ready n8n infrastructure for Hugging Face Spaces, including Docker setup, environment configuration, AI integrations (LangChain, OpenAI, Anthropic, Google Vertex AI), vector store (ChromaDB), Supabase PostgreSQL support, automated backup and restore scripts, and knowledge base synchronization. Includes documentation, security policy, and example configuration files.
- .DS_Store +0 -0
- LICENSE +171 -0
- README.md +458 -0
- SECURITY.md +147 -0
- config/.env.example +58 -0
- config/custom-nodes.json +44 -0
- docker/Dockerfile +56 -0
- docker/docker-compose.yml +101 -0
- knowledge/.DS_Store +0 -0
- package.json +48 -0
- scripts/backup.sh +198 -0
- scripts/restore.sh +204 -0
- scripts/sync-knowledge.sh +367 -0
- scripts/test-infrastructure.sh +190 -0
.DS_Store
ADDED
Binary file (6.15 kB). View file
|
|
LICENSE
ADDED
@@ -0,0 +1,171 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
Apache License
|
2 |
+
Version 2.0, January 2004
|
3 |
+
http://www.apache.org/licenses/
|
4 |
+
|
5 |
+
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
6 |
+
|
7 |
+
1. Definitions.
|
8 |
+
|
9 |
+
"License\" shall mean the terms and conditions for use, reproduction,
|
10 |
+
and distribution as defined by Sections 1 through 9 of this document.
|
11 |
+
|
12 |
+
"Licensor\" shall mean the copyright owner or entity granting the License.
|
13 |
+
|
14 |
+
"Legal Entity\" shall mean the union of the acting entity and all
|
15 |
+
other entities that control, are controlled by, or are under common
|
16 |
+
control with that entity. For the purposes of this definition,
|
17 |
+
"control\" means (i) the power, direct or indirect, to cause the
|
18 |
+
direction or management of such entity, whether by contract or
|
19 |
+
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
20 |
+
outstanding shares, or (iii) beneficial ownership of such entity.
|
21 |
+
|
22 |
+
"You" (or "Your") shall mean an individual or Legal Entity
|
23 |
+
exercising permissions granted by this License.
|
24 |
+
|
25 |
+
"Source\" form shall mean the preferred form for making modifications,
|
26 |
+
including but not limited to software source code, documentation
|
27 |
+
source, and configuration files.
|
28 |
+
|
29 |
+
"Object\" form shall mean any form resulting from mechanical
|
30 |
+
transformation or translation of a Source form, including but
|
31 |
+
not limited to compiled object code, generated documentation,
|
32 |
+
and conversions to other media types.
|
33 |
+
|
34 |
+
"Work\" shall mean the work of authorship, whether in Source or
|
35 |
+
Object form, made available under the License, as indicated by a
|
36 |
+
copyright notice that is included in or attached to the work
|
37 |
+
(which shall not include communications that are clearly marked or
|
38 |
+
otherwise designated in writing by the copyright owner as "Not a Work").
|
39 |
+
|
40 |
+
"Derivative Works\" shall mean any work, whether in Source or Object
|
41 |
+
form, that is based upon (or derived from) the Work and for which the
|
42 |
+
editorial revisions, annotations, elaborations, or other modifications
|
43 |
+
represent, as a whole, an original work of authorship. For the purposes
|
44 |
+
of this License, Derivative Works shall not include works that remain
|
45 |
+
separable from, or merely link (or bind by name) to the interfaces of,
|
46 |
+
the Work and derivative works thereof.
|
47 |
+
|
48 |
+
"Contribution\" shall mean any work of authorship, including
|
49 |
+
the original version of the Work and any modifications or additions
|
50 |
+
to that Work or Derivative Works thereof, that is intentionally
|
51 |
+
submitted to Licensor for inclusion in the Work by the copyright owner
|
52 |
+
or by an individual or Legal Entity authorized to submit on behalf of
|
53 |
+
the copyright owner. For the purposes of this definition, "submitted"
|
54 |
+
means any form of electronic, verbal, or written communication sent
|
55 |
+
to the Licensor or its representatives, including but not limited to
|
56 |
+
communication on electronic mailing lists, source code control
|
57 |
+
systems, and issue tracking systems that are managed by, or on behalf
|
58 |
+
of, the Licensor for the purpose of discussing and improving the Work,
|
59 |
+
but excluding communication that is conspicuously marked or otherwise
|
60 |
+
designated in writing by the copyright owner as "Not a Contribution."
|
61 |
+
|
62 |
+
2. Grant of Copyright License. Subject to the terms and conditions of
|
63 |
+
this License, each Contributor hereby grants to You a perpetual,
|
64 |
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
65 |
+
copyright license to use, reproduce, modify, distribute, and prepare
|
66 |
+
Derivative Works of, and to publicly perform and publicly display the
|
67 |
+
Work and such Derivative Works in Source or Object form.
|
68 |
+
|
69 |
+
3. Grant of Patent License. Subject to the terms and conditions of
|
70 |
+
this License, each Contributor hereby grants to You a perpetual,
|
71 |
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
72 |
+
(except as stated in this section) patent license to make, have made,
|
73 |
+
use, offer to sell, sell, import, and otherwise transfer the Work,
|
74 |
+
where such license applies only to those patent claims licensable
|
75 |
+
by such Contributor that are necessarily infringed by their
|
76 |
+
Contribution(s) alone or by combination of their Contribution(s)
|
77 |
+
with the Work to which such Contribution(s) was submitted. If You
|
78 |
+
institute patent litigation against any entity (including a
|
79 |
+
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
80 |
+
or a Contribution incorporated within the Work constitutes direct
|
81 |
+
or contributory patent infringement, then any patent licenses
|
82 |
+
granted to You under this License for that Work shall terminate
|
83 |
+
as of the date such litigation is filed.
|
84 |
+
|
85 |
+
4. Redistribution. You may reproduce and distribute copies of the
|
86 |
+
Work or Derivative Works thereof in any medium, with or without
|
87 |
+
modifications, and in Source or Object form, provided that You
|
88 |
+
meet the following conditions:
|
89 |
+
|
90 |
+
(a) You must give any other recipients of the Work or
|
91 |
+
Derivative Works a copy of this License; and
|
92 |
+
|
93 |
+
(b) You must cause any modified files to carry prominent notices
|
94 |
+
stating that You changed the files; and
|
95 |
+
|
96 |
+
(c) You must retain, in the Source form of any Derivative Works
|
97 |
+
that You distribute, all copyright, trademark, patent, attribution
|
98 |
+
notices from the Source form of the Work, excluding those notices
|
99 |
+
that do not pertain to any part of the Derivative Works; and
|
100 |
+
|
101 |
+
(d) If the Work includes a "NOTICE\" text file as part of its
|
102 |
+
distribution, then any Derivative Works that You distribute must
|
103 |
+
include a readable copy of the attribution notices contained
|
104 |
+
within such NOTICE file, excluding those notices that do not
|
105 |
+
pertain to any part of the Derivative Works, in at least one
|
106 |
+
of the following places: within a NOTICE text file distributed
|
107 |
+
as part of the Derivative Works; within the Source form or
|
108 |
+
documentation, if provided along with the Derivative Works; or,
|
109 |
+
within a display generated by the Derivative Works, if and
|
110 |
+
wherever such third-party notices normally appear. The contents
|
111 |
+
of the NOTICE file are for informational purposes only and
|
112 |
+
do not modify the License. You may add Your own attribution
|
113 |
+
notices within Derivative Works that You distribute, alongside
|
114 |
+
or as an addendum to the NOTICE text from the Work, provided
|
115 |
+
that such additional attribution notices cannot be construed
|
116 |
+
as modifying the License.
|
117 |
+
|
118 |
+
You may add Your own copyright notice to Your modifications and
|
119 |
+
may provide additional or different license terms and conditions
|
120 |
+
for use, reproduction, or distribution of Your modifications, or
|
121 |
+
for any such Derivative Works as a whole, provided Your use,
|
122 |
+
reproduction, and distribution of the Work otherwise complies with
|
123 |
+
the conditions stated in this License.
|
124 |
+
|
125 |
+
5. Submission of Contributions. Unless You explicitly state otherwise,
|
126 |
+
any Contribution intentionally submitted for inclusion in the Work
|
127 |
+
by You to the Licensor shall be under the terms and conditions of
|
128 |
+
this License, without any additional terms or conditions.
|
129 |
+
Notwithstanding the above, nothing herein shall supersede or modify
|
130 |
+
the terms of any separate license agreement you may have executed
|
131 |
+
with Licensor regarding such Contributions.
|
132 |
+
|
133 |
+
6. Trademarks. This License does not grant permission to use the trade
|
134 |
+
names, trademarks, service marks, or product names of the Licensor,
|
135 |
+
except as required for reasonable and customary use in describing the
|
136 |
+
origin of the Work and reproducing the content of the NOTICE file.
|
137 |
+
|
138 |
+
7. Disclaimer of Warranty. Unless required by applicable law or
|
139 |
+
agreed to in writing, Licensor provides the Work (and each
|
140 |
+
Contributor provides its Contributions) on an "AS IS\" BASIS,
|
141 |
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
142 |
+
implied, including, without limitation, any warranties or conditions
|
143 |
+
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
144 |
+
PARTICULAR PURPOSE. You are solely responsible for determining the
|
145 |
+
appropriateness of using or redistributing the Work and assume any
|
146 |
+
risks associated with Your exercise of permissions under this License.
|
147 |
+
|
148 |
+
8. Limitation of Liability. In no event and under no legal theory,
|
149 |
+
whether in tort (including negligence), contract, or otherwise,
|
150 |
+
unless required by applicable law (such as deliberate and grossly
|
151 |
+
negligent acts) or agreed to in writing, shall any Contributor be
|
152 |
+
liable to You for damages, including any direct, indirect, special,
|
153 |
+
incidental, or consequential damages of any character arising as a
|
154 |
+
result of this License or out of the use or inability to use the
|
155 |
+
Work (including but not limited to damages for loss of goodwill,
|
156 |
+
work stoppage, computer failure or malfunction, or any and all
|
157 |
+
other commercial damages or losses), even if such Contributor
|
158 |
+
has been advised of the possibility of such damages.
|
159 |
+
|
160 |
+
9. Accepting Warranty or Additional Liability. When redistributing
|
161 |
+
the Work or Derivative Works thereof, You may choose to offer,
|
162 |
+
and charge a fee for, acceptance of support, warranty, indemnity,
|
163 |
+
or other liability obligations and/or rights consistent with this
|
164 |
+
License. However, in accepting such obligations, You may act only
|
165 |
+
on Your own behalf and on Your sole responsibility, not on behalf
|
166 |
+
of any other Contributor, and only if You agree to indemnify,
|
167 |
+
defend, and hold each Contributor harmless for any liability
|
168 |
+
incurred by, or claims asserted against, such Contributor by reason
|
169 |
+
of your accepting any such warranty or additional liability.
|
170 |
+
|
171 |
+
END OF TERMS AND CONDITIONS
|
README.md
ADDED
@@ -0,0 +1,458 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
# n8n Infrastructure Repository
|
2 |
+
|
3 |
+
A comprehensive, production-ready infrastructure setup for deploying n8n automation platform on Hugging Face Spaces with AI integrations and automated knowledge management.
|
4 |
+
|
5 |
+
## 🚀 Features
|
6 |
+
|
7 |
+
### Core Platform
|
8 |
+
- **n8n v1.17.1**: Self-hosted workflow automation platform
|
9 |
+
- **Hugging Face Spaces**: Docker-based deployment with automatic scaling
|
10 |
+
- **Supabase PostgreSQL**: SSL-encrypted database with pgvector extension
|
11 |
+
- **ChromaDB**: Vector store for embeddings and AI-powered search
|
12 |
+
|
13 |
+
### AI & Automation
|
14 |
+
- **LangChain Integration**: Advanced AI workflow capabilities
|
15 |
+
- **Multi-Model Support**: OpenAI GPT, Anthropic Claude, Google Vertex AI
|
16 |
+
- **Vector Knowledge Base**: Automated content ingestion with embeddings
|
17 |
+
- **Community Nodes**: Extended functionality with custom AI nodes
|
18 |
+
|
19 |
+
### DevOps & Monitoring
|
20 |
+
- **GitHub Actions CI/CD**: Automated deployment and maintenance
|
21 |
+
- **Automated Backups**: Daily workflow and configuration backups
|
22 |
+
- **Knowledge Sync**: Multi-repository content synchronization
|
23 |
+
- **Health Monitoring**: Container health checks and alerting
|
24 |
+
|
25 |
+
## 📋 Prerequisites
|
26 |
+
|
27 |
+
Before setting up the infrastructure, ensure you have:
|
28 |
+
|
29 |
+
1. **GitHub Account** with repository access
|
30 |
+
2. **Hugging Face Account** with Spaces access
|
31 |
+
3. **Supabase Account** with PostgreSQL database
|
32 |
+
4. **Git** and **Docker** installed locally
|
33 |
+
|
34 |
+
### Required Secrets
|
35 |
+
|
36 |
+
Configure these secrets in your GitHub repository settings:
|
37 |
+
|
38 |
+
```bash
|
39 |
+
# Hugging Face
|
40 |
+
HF_USERNAME=your-huggingface-username
|
41 |
+
HF_TOKEN=your-hf-token
|
42 |
+
HF_SPACE_NAME=n8n-automation
|
43 |
+
|
44 |
+
# Database
|
45 |
+
DB_POSTGRESDB_HOST=your-project.supabase.co
|
46 |
+
DB_POSTGRESDB_USER=postgres
|
47 |
+
DB_POSTGRESDB_PASSWORD=your-database-password
|
48 |
+
DB_POSTGRESDB_DATABASE=postgres
|
49 |
+
|
50 |
+
# n8n Configuration
|
51 |
+
N8N_ENCRYPTION_KEY=your-32-character-encryption-key
|
52 |
+
N8N_USER_MANAGEMENT_JWT_SECRET=your-jwt-secret
|
53 |
+
WEBHOOK_URL=https://your-username-n8n-automation.hf.space
|
54 |
+
|
55 |
+
# AI Services (Optional)
|
56 |
+
OPENAI_API_KEY=sk-your-openai-key
|
57 |
+
ANTHROPIC_API_KEY=sk-ant-your-anthropic-key
|
58 |
+
GOOGLE_PROJECT_ID=your-gcp-project
|
59 |
+
```
|
60 |
+
|
61 |
+
## 🛠️ Quick Start
|
62 |
+
|
63 |
+
### 1. Repository Setup
|
64 |
+
|
65 |
+
```bash
|
66 |
+
# Clone the repository
|
67 |
+
git clone https://github.com/your-username/n8n-infra.git
|
68 |
+
cd n8n-infra
|
69 |
+
|
70 |
+
# Create environment configuration
|
71 |
+
cp config/.env.example .env
|
72 |
+
# Edit .env with your actual values
|
73 |
+
```
|
74 |
+
|
75 |
+
### 2. Local Development
|
76 |
+
|
77 |
+
```bash
|
78 |
+
# Start the full stack locally
|
79 |
+
docker-compose -f docker/docker-compose.yml up -d
|
80 |
+
|
81 |
+
# Check service status
|
82 |
+
docker-compose -f docker/docker-compose.yml ps
|
83 |
+
|
84 |
+
# View logs
|
85 |
+
docker-compose -f docker/docker-compose.yml logs -f n8n
|
86 |
+
```
|
87 |
+
|
88 |
+
### 3. Hugging Face Deployment
|
89 |
+
|
90 |
+
```bash
|
91 |
+
# Trigger deployment via GitHub Actions
|
92 |
+
git push origin main
|
93 |
+
|
94 |
+
# Or deploy manually
|
95 |
+
gh workflow run deploy-to-hf.yml
|
96 |
+
```
|
97 |
+
|
98 |
+
## 📊 Database Setup
|
99 |
+
|
100 |
+
### Supabase Configuration
|
101 |
+
|
102 |
+
1. **Create Supabase Project**:
|
103 |
+
```sql
|
104 |
+
-- Enable pgvector extension
|
105 |
+
CREATE EXTENSION IF NOT EXISTS vector;
|
106 |
+
|
107 |
+
-- Create knowledge base schema
|
108 |
+
CREATE SCHEMA IF NOT EXISTS knowledge;
|
109 |
+
|
110 |
+
-- Create embeddings table
|
111 |
+
CREATE TABLE knowledge.embeddings (
|
112 |
+
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
113 |
+
content_id TEXT NOT NULL,
|
114 |
+
collection_name TEXT NOT NULL,
|
115 |
+
content TEXT NOT NULL,
|
116 |
+
embedding VECTOR(384),
|
117 |
+
metadata JSONB DEFAULT '{}',
|
118 |
+
created_at TIMESTAMPTZ DEFAULT NOW(),
|
119 |
+
updated_at TIMESTAMPTZ DEFAULT NOW()
|
120 |
+
);
|
121 |
+
|
122 |
+
-- Create indexes for performance
|
123 |
+
CREATE INDEX IF NOT EXISTS idx_embeddings_collection ON knowledge.embeddings(collection_name);
|
124 |
+
CREATE INDEX IF NOT EXISTS idx_embeddings_content_id ON knowledge.embeddings(content_id);
|
125 |
+
CREATE INDEX IF NOT EXISTS idx_embeddings_vector ON knowledge.embeddings
|
126 |
+
USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100);
|
127 |
+
```
|
128 |
+
|
129 |
+
2. **Configure Row Level Security**:
|
130 |
+
```sql
|
131 |
+
-- Enable RLS
|
132 |
+
ALTER TABLE knowledge.embeddings ENABLE ROW LEVEL SECURITY;
|
133 |
+
|
134 |
+
-- Allow authenticated users to read embeddings
|
135 |
+
CREATE POLICY "Users can read embeddings" ON knowledge.embeddings
|
136 |
+
FOR SELECT TO authenticated USING (true);
|
137 |
+
|
138 |
+
-- Allow service role to manage embeddings
|
139 |
+
CREATE POLICY "Service role can manage embeddings" ON knowledge.embeddings
|
140 |
+
FOR ALL TO service_role USING (true);
|
141 |
+
```
|
142 |
+
|
143 |
+
## 🤖 AI Integration Guide
|
144 |
+
|
145 |
+
### LangChain Workflows
|
146 |
+
|
147 |
+
The platform supports advanced LangChain workflows:
|
148 |
+
|
149 |
+
```javascript
|
150 |
+
// Example: Knowledge-based Q&A workflow
|
151 |
+
{
|
152 |
+
"nodes": [
|
153 |
+
{
|
154 |
+
"name": "Vector Search",
|
155 |
+
"type": "n8n-nodes-vector-store",
|
156 |
+
"parameters": {
|
157 |
+
"operation": "similarity_search",
|
158 |
+
"query": "{{ $json.question }}",
|
159 |
+
"collection": "n8n",
|
160 |
+
"top_k": 5
|
161 |
+
}
|
162 |
+
},
|
163 |
+
{
|
164 |
+
"name": "LangChain QA",
|
165 |
+
"type": "@n8n/n8n-nodes-langchain",
|
166 |
+
"parameters": {
|
167 |
+
"chain_type": "question_answering",
|
168 |
+
"context": "{{ $json.vector_results }}",
|
169 |
+
"question": "{{ $json.question }}"
|
170 |
+
}
|
171 |
+
}
|
172 |
+
]
|
173 |
+
}
|
174 |
+
```
|
175 |
+
|
176 |
+
### Custom AI Nodes
|
177 |
+
|
178 |
+
Install additional AI nodes:
|
179 |
+
|
180 |
+
```bash
|
181 |
+
# Install in running container
|
182 |
+
docker exec n8n-automation npm install n8n-nodes-google-vertex-ai
|
183 |
+
docker exec n8n-automation npm install n8n-nodes-openai-advanced
|
184 |
+
|
185 |
+
# Restart to load new nodes
|
186 |
+
docker-compose -f docker/docker-compose.yml restart n8n
|
187 |
+
```
|
188 |
+
|
189 |
+
## 🗄️ Knowledge Management
|
190 |
+
|
191 |
+
### Automated Synchronization
|
192 |
+
|
193 |
+
The system automatically syncs content from these repositories:
|
194 |
+
|
195 |
+
- **n8n Knowledge**: `/projects/n8n` - Workflow examples and best practices
|
196 |
+
- **Video & Animation**: `/projects/videos-e-animacoes` - Multimedia processing guides
|
197 |
+
- **Midjourney Prompts**: `/projects/midjorney-prompt` - AI art generation prompts
|
198 |
+
|
199 |
+
### Manual Knowledge Sync
|
200 |
+
|
201 |
+
```bash
|
202 |
+
# Sync specific collection
|
203 |
+
./scripts/sync-knowledge.sh
|
204 |
+
|
205 |
+
# Or trigger via GitHub Actions
|
206 |
+
gh workflow run sync-knowledge.yml -f collections=n8n,midjourney-prompt
|
207 |
+
```
|
208 |
+
|
209 |
+
### Vector Search Setup
|
210 |
+
|
211 |
+
Query the knowledge base in n8n workflows:
|
212 |
+
|
213 |
+
```javascript
|
214 |
+
// Vector similarity search node configuration
|
215 |
+
{
|
216 |
+
"collection": "n8n",
|
217 |
+
"query": "How to create webhook workflows",
|
218 |
+
"top_k": 3,
|
219 |
+
"score_threshold": 0.7
|
220 |
+
}
|
221 |
+
```
|
222 |
+
|
223 |
+
## 💾 Backup & Recovery
|
224 |
+
|
225 |
+
### Automated Backups
|
226 |
+
|
227 |
+
Daily backups include:
|
228 |
+
- All n8n workflows (exported as JSON)
|
229 |
+
- Encrypted credentials
|
230 |
+
- Database schema
|
231 |
+
- Knowledge base content
|
232 |
+
- Vector embeddings
|
233 |
+
|
234 |
+
### Manual Backup
|
235 |
+
|
236 |
+
```bash
|
237 |
+
# Create full backup
|
238 |
+
./scripts/backup.sh custom-backup-name
|
239 |
+
|
240 |
+
# List available backups
|
241 |
+
ls workflows/backup/
|
242 |
+
|
243 |
+
# Restore from backup
|
244 |
+
./scripts/restore.sh n8n_backup_20240115_140230
|
245 |
+
```
|
246 |
+
|
247 |
+
### Backup Schedule
|
248 |
+
|
249 |
+
- **Daily**: Automated workflow backup at 2 AM UTC
|
250 |
+
- **Weekly**: Full system backup including database
|
251 |
+
- **On-demand**: Manual backups via GitHub Actions
|
252 |
+
|
253 |
+
## 🔧 Maintenance
|
254 |
+
|
255 |
+
### Health Monitoring
|
256 |
+
|
257 |
+
```bash
|
258 |
+
# Check container health
|
259 |
+
docker-compose -f docker/docker-compose.yml ps
|
260 |
+
|
261 |
+
# View application logs
|
262 |
+
docker-compose -f docker/docker-compose.yml logs -f n8n
|
263 |
+
|
264 |
+
# Monitor vector store
|
265 |
+
curl http://localhost:8000/api/v1/heartbeat
|
266 |
+
```
|
267 |
+
|
268 |
+
### Performance Tuning
|
269 |
+
|
270 |
+
**Database Optimization**:
|
271 |
+
```sql
|
272 |
+
-- Monitor query performance
|
273 |
+
SELECT query, mean_exec_time, calls
|
274 |
+
FROM pg_stat_statements
|
275 |
+
WHERE query LIKE '%n8n%'
|
276 |
+
ORDER BY mean_exec_time DESC
|
277 |
+
LIMIT 10;
|
278 |
+
|
279 |
+
-- Optimize vector searches
|
280 |
+
SET ivfflat.probes = 10;
|
281 |
+
```
|
282 |
+
|
283 |
+
**Container Resources**:
|
284 |
+
```yaml
|
285 |
+
# docker-compose.yml resource limits
|
286 |
+
services:
|
287 |
+
n8n:
|
288 |
+
deploy:
|
289 |
+
resources:
|
290 |
+
limits:
|
291 |
+
cpus: '2.0'
|
292 |
+
memory: 4G
|
293 |
+
reservations:
|
294 |
+
cpus: '1.0'
|
295 |
+
memory: 2G
|
296 |
+
```
|
297 |
+
|
298 |
+
## 🔒 Security
|
299 |
+
|
300 |
+
### SSL Configuration
|
301 |
+
|
302 |
+
- All database connections use SSL encryption
|
303 |
+
- Webhook URLs must use HTTPS
|
304 |
+
- Container communication over encrypted networks
|
305 |
+
|
306 |
+
### Credential Management
|
307 |
+
|
308 |
+
```bash
|
309 |
+
# Credentials are encrypted by n8n
|
310 |
+
# Store sensitive files in config/credentials/
|
311 |
+
mkdir -p config/credentials
|
312 |
+
echo '{}' > config/credentials/google-service-account.json
|
313 |
+
|
314 |
+
# Set proper permissions
|
315 |
+
chmod 600 config/credentials/*
|
316 |
+
```
|
317 |
+
|
318 |
+
### Environment Security
|
319 |
+
|
320 |
+
- Never commit `.env` files
|
321 |
+
- Use GitHub Secrets for sensitive data
|
322 |
+
- Rotate encryption keys regularly
|
323 |
+
- Enable Supabase RLS policies
|
324 |
+
|
325 |
+
## 🚨 Troubleshooting
|
326 |
+
|
327 |
+
### Common Issues
|
328 |
+
|
329 |
+
**Connection Problems**:
|
330 |
+
```bash
|
331 |
+
# Test database connection
|
332 |
+
docker exec n8n-automation psql "$DB_POSTGRESDB_HOST" -U "$DB_POSTGRESDB_USER" -c "\l"
|
333 |
+
|
334 |
+
# Check n8n logs
|
335 |
+
docker logs n8n-automation --tail 50
|
336 |
+
|
337 |
+
# Verify webhook connectivity
|
338 |
+
curl -I "$WEBHOOK_URL/healthz"
|
339 |
+
```
|
340 |
+
|
341 |
+
**Deployment Issues**:
|
342 |
+
```bash
|
343 |
+
# Check Hugging Face Space status
|
344 |
+
curl -I "https://huggingface.co/spaces/$HF_USERNAME/$HF_SPACE_NAME"
|
345 |
+
|
346 |
+
# View GitHub Actions logs
|
347 |
+
gh run list --workflow=deploy-to-hf.yml
|
348 |
+
gh run view [run-id] --log
|
349 |
+
```
|
350 |
+
|
351 |
+
**Knowledge Sync Problems**:
|
352 |
+
```bash
|
353 |
+
# Manual knowledge sync debug
|
354 |
+
./scripts/sync-knowledge.sh
|
355 |
+
echo $? # Should return 0 for success
|
356 |
+
|
357 |
+
# Check embedding generation
|
358 |
+
python3 -c "
|
359 |
+
import json
|
360 |
+
with open('knowledge/n8n/n8n_embeddings.json') as f:
|
361 |
+
data = json.load(f)
|
362 |
+
print(f'Embeddings loaded: {len(data)} documents')
|
363 |
+
"
|
364 |
+
```
|
365 |
+
|
366 |
+
### Recovery Procedures
|
367 |
+
|
368 |
+
**Emergency Restore**:
|
369 |
+
1. Stop all services: `docker-compose down`
|
370 |
+
2. Restore from latest backup: `./scripts/restore.sh [backup-name]`
|
371 |
+
3. Restart services: `docker-compose up -d`
|
372 |
+
4. Verify functionality: Access web interface
|
373 |
+
|
374 |
+
**Database Recovery**:
|
375 |
+
```sql
|
376 |
+
-- Check database integrity
|
377 |
+
SELECT schemaname, tablename, n_tup_ins, n_tup_upd, n_tup_del
|
378 |
+
FROM pg_stat_user_tables
|
379 |
+
WHERE schemaname = 'public';
|
380 |
+
|
381 |
+
-- Rebuild vector indexes if needed
|
382 |
+
REINDEX INDEX idx_embeddings_vector;
|
383 |
+
```
|
384 |
+
|
385 |
+
## 📈 Scaling
|
386 |
+
|
387 |
+
### Horizontal Scaling
|
388 |
+
|
389 |
+
- Multiple n8n instances with queue mode
|
390 |
+
- Load balancer configuration
|
391 |
+
- Distributed vector store
|
392 |
+
|
393 |
+
### Performance Monitoring
|
394 |
+
|
395 |
+
- Enable n8n metrics: `N8N_METRICS=true`
|
396 |
+
- Database query monitoring
|
397 |
+
- Vector search performance tracking
|
398 |
+
- Container resource utilization
|
399 |
+
|
400 |
+
## 🔄 CI/CD Pipeline
|
401 |
+
|
402 |
+
### Workflow Triggers
|
403 |
+
|
404 |
+
- **Push to main**: Automatic deployment
|
405 |
+
- **Scheduled**: Daily backups and knowledge sync
|
406 |
+
- **Manual**: On-demand operations via GitHub Actions
|
407 |
+
|
408 |
+
### Pipeline Stages
|
409 |
+
|
410 |
+
1. **Build**: Docker image creation and testing
|
411 |
+
2. **Test**: Health checks and validation
|
412 |
+
3. **Deploy**: Hugging Face Spaces deployment
|
413 |
+
4. **Monitor**: Post-deployment verification
|
414 |
+
|
415 |
+
## 📝 Contributing
|
416 |
+
|
417 |
+
1. Fork the repository
|
418 |
+
2. Create feature branch: `git checkout -b feature/new-capability`
|
419 |
+
3. Commit changes: `git commit -am 'Add new capability'`
|
420 |
+
4. Push branch: `git push origin feature/new-capability`
|
421 |
+
5. Submit pull request
|
422 |
+
|
423 |
+
### Development Workflow
|
424 |
+
|
425 |
+
```bash
|
426 |
+
# Local development
|
427 |
+
docker-compose -f docker/docker-compose.yml up --build
|
428 |
+
|
429 |
+
# Run tests
|
430 |
+
./scripts/test-infrastructure.sh
|
431 |
+
|
432 |
+
# Deploy to staging
|
433 |
+
gh workflow run deploy-to-hf.yml -f force_deploy=true
|
434 |
+
```
|
435 |
+
|
436 |
+
## 📞 Support
|
437 |
+
|
438 |
+
- **Issues**: GitHub Issues
|
439 |
+
- **Documentation**: [n8n Documentation](https://docs.n8n.io)
|
440 |
+
- **Community**: [n8n Community](https://community.n8n.io)
|
441 |
+
|
442 |
+
## 📄 License
|
443 |
+
|
444 |
+
This project is licensed under the Apache License 2.0 - see the LICENSE file for details.
|
445 |
+
|
446 |
+
---
|
447 |
+
|
448 |
+
**⚡ Pro Tips**:
|
449 |
+
|
450 |
+
1. **Performance**: Use queue mode for high-volume workflows
|
451 |
+
2. **Security**: Regular credential rotation and access reviews
|
452 |
+
3. **Monitoring**: Set up alerts for failed workflows and system health
|
453 |
+
4. **Backup**: Test restore procedures regularly
|
454 |
+
5. **Knowledge**: Keep your knowledge base updated with latest best practices
|
455 |
+
|
456 |
+
---
|
457 |
+
|
458 |
+
*Built with ❤️ for the n8n automation community*
|
SECURITY.md
ADDED
@@ -0,0 +1,147 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
# Security Policy
|
2 |
+
|
3 |
+
## Supported Versions
|
4 |
+
|
5 |
+
We actively maintain and provide security updates for the following versions:
|
6 |
+
|
7 |
+
| Version | Supported |
|
8 |
+
| ------- | ------------------ |
|
9 |
+
| 1.0.x | :white_check_mark: |
|
10 |
+
|
11 |
+
## Reporting a Vulnerability
|
12 |
+
|
13 |
+
We take security vulnerabilities seriously. If you discover a security vulnerability in this n8n infrastructure, please report it responsibly.
|
14 |
+
|
15 |
+
### How to Report
|
16 |
+
|
17 |
+
1. **Do NOT** open a public GitHub issue for security vulnerabilities
|
18 |
+
2. Send an email to: [email protected] (replace with your actual security contact)
|
19 |
+
3. Include the following information:
|
20 |
+
- Description of the vulnerability
|
21 |
+
- Steps to reproduce the issue
|
22 |
+
- Potential impact assessment
|
23 |
+
- Suggested fix (if available)
|
24 |
+
|
25 |
+
### What to Expect
|
26 |
+
|
27 |
+
- **Acknowledgment**: Within 48 hours of your report
|
28 |
+
- **Initial Assessment**: Within 7 days
|
29 |
+
- **Fix Timeline**: Critical issues within 14 days, others within 30 days
|
30 |
+
- **Disclosure**: Coordinated disclosure after fix is released
|
31 |
+
|
32 |
+
## Security Best Practices
|
33 |
+
|
34 |
+
### For Administrators
|
35 |
+
|
36 |
+
1. **Environment Variables**:
|
37 |
+
- Never commit `.env` files to version control
|
38 |
+
- Use GitHub Secrets for all sensitive data
|
39 |
+
- Rotate encryption keys regularly (quarterly recommended)
|
40 |
+
|
41 |
+
2. **Database Security**:
|
42 |
+
- Always use SSL connections to Supabase
|
43 |
+
- Enable Row Level Security (RLS) policies
|
44 |
+
- Regular backup encryption validation
|
45 |
+
- Monitor for unusual database activity
|
46 |
+
|
47 |
+
3. **Container Security**:
|
48 |
+
- Keep n8n version pinned and updated
|
49 |
+
- Regular security scanning of Docker images
|
50 |
+
- Use non-root user inside containers
|
51 |
+
- Limit container network access
|
52 |
+
|
53 |
+
4. **Access Control**:
|
54 |
+
- Enable n8n user management
|
55 |
+
- Use strong JWT secrets
|
56 |
+
- Implement webhook authentication
|
57 |
+
- Regular access review and cleanup
|
58 |
+
|
59 |
+
### For Developers
|
60 |
+
|
61 |
+
1. **Code Security**:
|
62 |
+
- No hardcoded credentials in source code
|
63 |
+
- Validate all webhook inputs
|
64 |
+
- Sanitize user inputs in workflows
|
65 |
+
- Use prepared statements for database queries
|
66 |
+
|
67 |
+
2. **Workflow Security**:
|
68 |
+
- Audit workflow permissions regularly
|
69 |
+
- Secure credential storage in n8n
|
70 |
+
- Validate external API responses
|
71 |
+
- Implement proper error handling
|
72 |
+
|
73 |
+
3. **AI Integration Security**:
|
74 |
+
- Validate AI model outputs
|
75 |
+
- Sanitize prompts and inputs
|
76 |
+
- Secure API key management
|
77 |
+
- Monitor AI usage and costs
|
78 |
+
|
79 |
+
## Security Checklist
|
80 |
+
|
81 |
+
### Pre-Deployment
|
82 |
+
- [ ] All secrets configured in GitHub repository
|
83 |
+
- [ ] Database SSL enforcement enabled
|
84 |
+
- [ ] Container security scan passed
|
85 |
+
- [ ] Webhook authentication configured
|
86 |
+
- [ ] Network security policies reviewed
|
87 |
+
|
88 |
+
### Post-Deployment
|
89 |
+
- [ ] Health monitoring enabled
|
90 |
+
- [ ] Backup encryption verified
|
91 |
+
- [ ] Access logs configured
|
92 |
+
- [ ] Incident response plan ready
|
93 |
+
- [ ] Security contact information updated
|
94 |
+
|
95 |
+
### Regular Maintenance
|
96 |
+
- [ ] Monthly security updates applied
|
97 |
+
- [ ] Quarterly credential rotation
|
98 |
+
- [ ] Backup integrity verification
|
99 |
+
- [ ] Security audit review
|
100 |
+
- [ ] Vulnerability scanning
|
101 |
+
|
102 |
+
## Known Security Considerations
|
103 |
+
|
104 |
+
1. **Hugging Face Spaces**: Public spaces expose the application URL. Use authentication and access controls.
|
105 |
+
|
106 |
+
2. **Vector Embeddings**: Knowledge base content may contain sensitive information. Review before indexing.
|
107 |
+
|
108 |
+
3. **Webhook Endpoints**: Publicly accessible URLs should implement proper authentication.
|
109 |
+
|
110 |
+
4. **Database Access**: Ensure Supabase RLS policies are properly configured for your use case.
|
111 |
+
|
112 |
+
## Incident Response
|
113 |
+
|
114 |
+
In case of a security incident:
|
115 |
+
|
116 |
+
1. **Immediate Actions**:
|
117 |
+
- Disable affected services if necessary
|
118 |
+
- Preserve logs and evidence
|
119 |
+
- Assess scope and impact
|
120 |
+
|
121 |
+
2. **Communication**:
|
122 |
+
- Notify security team immediately
|
123 |
+
- Prepare user communication if needed
|
124 |
+
- Coordinate with stakeholders
|
125 |
+
|
126 |
+
3. **Recovery**:
|
127 |
+
- Apply security patches
|
128 |
+
- Restore from clean backups if needed
|
129 |
+
- Verify system integrity
|
130 |
+
- Update security measures
|
131 |
+
|
132 |
+
## Security Resources
|
133 |
+
|
134 |
+
- [n8n Security Documentation](https://docs.n8n.io/security/)
|
135 |
+
- [Supabase Security Guide](https://supabase.com/docs/guides/platform/security)
|
136 |
+
- [Docker Security Best Practices](https://docs.docker.com/develop/security-best-practices/)
|
137 |
+
- [GitHub Actions Security](https://docs.github.com/en/actions/security-guides)
|
138 |
+
|
139 |
+
## Contact
|
140 |
+
|
141 |
+
For security-related questions or concerns:
|
142 |
+
- Email: [email protected]
|
143 |
+
- Security Team: @security-team (GitHub)
|
144 |
+
|
145 |
+
---
|
146 |
+
|
147 |
+
*Last updated: January 2025*
|
config/.env.example
ADDED
@@ -0,0 +1,58 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
# n8n Infrastructure Environment Configuration
|
2 |
+
# Copy this file to .env and fill in your actual values
|
3 |
+
|
4 |
+
# ===== CORE N8N CONFIGURATION =====
|
5 |
+
N8N_ENCRYPTION_KEY=your-32-character-encryption-key-here
|
6 |
+
N8N_USER_MANAGEMENT_JWT_SECRET=your-jwt-secret-key-here
|
7 |
+
N8N_HOST=your-n8n-domain.com
|
8 |
+
WEBHOOK_URL=https://your-n8n-domain.com
|
9 |
+
|
10 |
+
# ===== DATABASE CONFIGURATION =====
|
11 |
+
DB_TYPE=postgresdb
|
12 |
+
DB_POSTGRESDB_HOST=your-supabase-host.supabase.co
|
13 |
+
DB_POSTGRESDB_PORT=5432
|
14 |
+
DB_POSTGRESDB_DATABASE=postgres
|
15 |
+
DB_POSTGRESDB_USER=postgres
|
16 |
+
DB_POSTGRESDB_PASSWORD=your-supabase-password
|
17 |
+
DB_POSTGRESDB_SSL=true
|
18 |
+
|
19 |
+
# ===== DEPLOYMENT CONFIGURATION =====
|
20 |
+
HF_TOKEN=your-hugging-face-token
|
21 |
+
HF_SPACE_NAME=your-username/n8n-automation
|
22 |
+
GITHUB_TOKEN=your-github-token
|
23 |
+
|
24 |
+
# ===== AI INTEGRATIONS =====
|
25 |
+
GOOGLE_PROJECT_ID=your-google-cloud-project-id
|
26 |
+
GOOGLE_CREDENTIALS_PATH=/home/node/.n8n/credentials/google-service-account.json
|
27 |
+
OPENAI_API_KEY=sk-your-openai-api-key
|
28 |
+
ANTHROPIC_API_KEY=sk-ant-your-anthropic-key
|
29 |
+
VERTEX_AI_PROJECT=your-vertex-ai-project
|
30 |
+
VERTEX_AI_LOCATION=us-central1
|
31 |
+
|
32 |
+
# ===== VECTOR STORE CONFIGURATION =====
|
33 |
+
CHROMA_AUTH_TOKEN=your-chroma-auth-token
|
34 |
+
CHROMA_HOST=vector-store
|
35 |
+
CHROMA_PORT=8000
|
36 |
+
|
37 |
+
# ===== KNOWLEDGE BASE SYNC =====
|
38 |
+
KB_REPO_N8N=https://github.com/danilonovaisv/CHATGPT-knowledge-base.git
|
39 |
+
KB_BRANCH_N8N=main
|
40 |
+
KB_PATH_N8N=projects/n8n
|
41 |
+
KB_PATH_VIDEOS=projects/videos-e-animacoes
|
42 |
+
KB_PATH_MIDJOURNEY=projects/midjorney-prompt
|
43 |
+
|
44 |
+
# ===== MONITORING AND LOGGING =====
|
45 |
+
N8N_LOG_LEVEL=info
|
46 |
+
N8N_METRICS=true
|
47 |
+
SENTRY_DSN=your-sentry-dsn (optional)
|
48 |
+
|
49 |
+
# ===== BACKUP CONFIGURATION =====
|
50 |
+
BACKUP_SCHEDULE=0 2 * * *
|
51 |
+
BACKUP_RETENTION_DAYS=30
|
52 |
+
BACKUP_ENCRYPTION_PASSWORD=your-backup-encryption-password
|
53 |
+
|
54 |
+
# ===== SECURITY =====
|
55 |
+
ALLOWED_ORIGINS=https://your-domain.com,https://your-n8n-domain.com
|
56 |
+
CSRF_SECRET=your-csrf-secret
|
57 |
+
RATE_LIMIT_WINDOW=15
|
58 |
+
RATE_LIMIT_MAX=100
|
config/custom-nodes.json
ADDED
@@ -0,0 +1,44 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
{
|
2 |
+
"name": "n8n-custom-configuration",
|
3 |
+
"version": "1.0.0",
|
4 |
+
"description": "Custom configuration for n8n automation platform",
|
5 |
+
"customNodes": [
|
6 |
+
{
|
7 |
+
"name": "@n8n/n8n-nodes-langchain",
|
8 |
+
"version": "latest",
|
9 |
+
"description": "LangChain integration nodes for AI workflows"
|
10 |
+
},
|
11 |
+
{
|
12 |
+
"name": "n8n-nodes-google-vertex-ai",
|
13 |
+
"version": "latest",
|
14 |
+
"description": "Google Vertex AI integration"
|
15 |
+
},
|
16 |
+
{
|
17 |
+
"name": "n8n-nodes-supabase",
|
18 |
+
"version": "latest",
|
19 |
+
"description": "Supabase database operations"
|
20 |
+
},
|
21 |
+
{
|
22 |
+
"name": "n8n-nodes-vector-store",
|
23 |
+
"version": "latest",
|
24 |
+
"description": "Vector database operations for embeddings"
|
25 |
+
}
|
26 |
+
],
|
27 |
+
"credentialTypes": [
|
28 |
+
{
|
29 |
+
"name": "googleCloudApi",
|
30 |
+
"displayName": "Google Cloud API",
|
31 |
+
"documentationUrl": "https://cloud.google.com/docs/authentication"
|
32 |
+
},
|
33 |
+
{
|
34 |
+
"name": "supabaseApi",
|
35 |
+
"displayName": "Supabase API",
|
36 |
+
"documentationUrl": "https://supabase.com/docs/guides/api"
|
37 |
+
},
|
38 |
+
{
|
39 |
+
"name": "openAiApi",
|
40 |
+
"displayName": "OpenAI API",
|
41 |
+
"documentationUrl": "https://platform.openai.com/docs/api-reference"
|
42 |
+
}
|
43 |
+
]
|
44 |
+
}
|
docker/Dockerfile
ADDED
@@ -0,0 +1,56 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
# Production-ready n8n Dockerfile for Hugging Face Spaces
|
2 |
+
FROM n8nio/n8n:1.17.1
|
3 |
+
|
4 |
+
# Set environment for production
|
5 |
+
ENV NODE_ENV=production
|
6 |
+
ENV N8N_PORT=7860
|
7 |
+
ENV N8N_LISTEN_ADDRESS=0.0.0.0
|
8 |
+
ENV N8N_PROTOCOL=https
|
9 |
+
ENV N8N_METRICS=true
|
10 |
+
ENV N8N_LOG_LEVEL=info
|
11 |
+
ENV N8N_LOG_OUTPUT=console
|
12 |
+
ENV EXECUTIONS_MODE=queue
|
13 |
+
ENV N8N_DISABLE_UI=false
|
14 |
+
|
15 |
+
# Create app user for security
|
16 |
+
USER root
|
17 |
+
|
18 |
+
# Install additional dependencies for AI and automation
|
19 |
+
RUN apk add --no-cache \
|
20 |
+
curl \
|
21 |
+
git \
|
22 |
+
python3 \
|
23 |
+
py3-pip \
|
24 |
+
postgresql-client \
|
25 |
+
&& pip3 install --no-cache-dir \
|
26 |
+
langchain \
|
27 |
+
chromadb \
|
28 |
+
sentence-transformers
|
29 |
+
|
30 |
+
# Create necessary directories
|
31 |
+
RUN mkdir -p /home/node/.n8n/custom \
|
32 |
+
&& mkdir -p /home/node/.n8n/nodes \
|
33 |
+
&& mkdir -p /home/node/.n8n/credentials \
|
34 |
+
&& mkdir -p /home/node/knowledge
|
35 |
+
|
36 |
+
# Copy custom configurations
|
37 |
+
COPY config/custom-nodes.json /home/node/.n8n/custom/
|
38 |
+
COPY knowledge/ /home/node/knowledge/
|
39 |
+
|
40 |
+
# Set proper permissions
|
41 |
+
RUN chown -R node:node /home/node/.n8n \
|
42 |
+
&& chown -R node:node /home/node/knowledge \
|
43 |
+
&& chmod +x /home/node/knowledge/sync-*.sh
|
44 |
+
|
45 |
+
# Switch back to node user
|
46 |
+
USER node
|
47 |
+
|
48 |
+
# Health check
|
49 |
+
HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 \
|
50 |
+
CMD curl -f http://localhost:7860/healthz || exit 1
|
51 |
+
|
52 |
+
# Expose port for Hugging Face Spaces
|
53 |
+
EXPOSE 7860
|
54 |
+
|
55 |
+
# Start n8n
|
56 |
+
CMD ["n8n", "start"]
|
docker/docker-compose.yml
ADDED
@@ -0,0 +1,101 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
version: '3.8'
|
2 |
+
|
3 |
+
services:
|
4 |
+
n8n:
|
5 |
+
build:
|
6 |
+
context: ..
|
7 |
+
dockerfile: docker/Dockerfile
|
8 |
+
container_name: n8n-automation
|
9 |
+
restart: unless-stopped
|
10 |
+
ports:
|
11 |
+
- "7860:7860"
|
12 |
+
environment:
|
13 |
+
# Core Configuration
|
14 |
+
- NODE_ENV=production
|
15 |
+
- N8N_PORT=7860
|
16 |
+
- N8N_LISTEN_ADDRESS=0.0.0.0
|
17 |
+
- N8N_PROTOCOL=https
|
18 |
+
- N8N_HOST=${N8N_HOST:-localhost}
|
19 |
+
|
20 |
+
# Security
|
21 |
+
- N8N_ENCRYPTION_KEY=${N8N_ENCRYPTION_KEY}
|
22 |
+
- N8N_USER_MANAGEMENT_JWT_SECRET=${N8N_USER_MANAGEMENT_JWT_SECRET}
|
23 |
+
|
24 |
+
# Database Configuration
|
25 |
+
- DB_TYPE=postgresdb
|
26 |
+
- DB_POSTGRESDB_HOST=${DB_POSTGRESDB_HOST}
|
27 |
+
- DB_POSTGRESDB_PORT=${DB_POSTGRESDB_PORT:-5432}
|
28 |
+
- DB_POSTGRESDB_DATABASE=${DB_POSTGRESDB_DATABASE}
|
29 |
+
- DB_POSTGRESDB_USER=${DB_POSTGRESDB_USER}
|
30 |
+
- DB_POSTGRESDB_PASSWORD=${DB_POSTGRESDB_PASSWORD}
|
31 |
+
- DB_POSTGRESDB_SSL=true
|
32 |
+
|
33 |
+
# Webhooks and External Access
|
34 |
+
- WEBHOOK_URL=${WEBHOOK_URL}
|
35 |
+
- N8N_EDITOR_BASE_URL=${WEBHOOK_URL}
|
36 |
+
|
37 |
+
# AI and External Integrations
|
38 |
+
- GOOGLE_PROJECT_ID=${GOOGLE_PROJECT_ID}
|
39 |
+
- GOOGLE_CREDENTIALS_PATH=${GOOGLE_CREDENTIALS_PATH}
|
40 |
+
- OPENAI_API_KEY=${OPENAI_API_KEY}
|
41 |
+
- ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
|
42 |
+
|
43 |
+
# Performance
|
44 |
+
- EXECUTIONS_MODE=queue
|
45 |
+
- EXECUTIONS_DATA_SAVE_ON_ERROR=all
|
46 |
+
- EXECUTIONS_DATA_SAVE_ON_SUCCESS=all
|
47 |
+
- EXECUTIONS_DATA_SAVE_MANUAL_EXECUTIONS=true
|
48 |
+
|
49 |
+
# Logging and Monitoring
|
50 |
+
- N8N_LOG_LEVEL=info
|
51 |
+
- N8N_LOG_OUTPUT=console
|
52 |
+
- N8N_METRICS=true
|
53 |
+
|
54 |
+
volumes:
|
55 |
+
- n8n_data:/home/node/.n8n
|
56 |
+
- ./knowledge:/home/node/knowledge:ro
|
57 |
+
- ./config/credentials:/home/node/.n8n/credentials:ro
|
58 |
+
- ./workflows/backup:/home/node/.n8n/backup
|
59 |
+
|
60 |
+
networks:
|
61 |
+
- n8n-network
|
62 |
+
|
63 |
+
depends_on:
|
64 |
+
- vector-store
|
65 |
+
- backup-service
|
66 |
+
|
67 |
+
vector-store:
|
68 |
+
image: chromadb/chroma:latest
|
69 |
+
container_name: n8n-vector-store
|
70 |
+
restart: unless-stopped
|
71 |
+
ports:
|
72 |
+
- "8000:8000"
|
73 |
+
environment:
|
74 |
+
- CHROMA_SERVER_AUTHN_CREDENTIALS=${CHROMA_AUTH_TOKEN}
|
75 |
+
- CHROMA_SERVER_AUTHN_PROVIDER=chromadb.auth.token.TokenAuthServerProvider
|
76 |
+
volumes:
|
77 |
+
- vector_data:/chroma/chroma
|
78 |
+
networks:
|
79 |
+
- n8n-network
|
80 |
+
|
81 |
+
backup-service:
|
82 |
+
image: alpine:latest
|
83 |
+
container_name: n8n-backup
|
84 |
+
restart: "no"
|
85 |
+
command: /bin/sh -c "while true; do sleep 3600; done"
|
86 |
+
volumes:
|
87 |
+
- n8n_data:/backup/n8n:ro
|
88 |
+
- ./scripts:/scripts:ro
|
89 |
+
- ./workflows/backup:/backup/workflows
|
90 |
+
networks:
|
91 |
+
- n8n-network
|
92 |
+
|
93 |
+
volumes:
|
94 |
+
n8n_data:
|
95 |
+
driver: local
|
96 |
+
vector_data:
|
97 |
+
driver: local
|
98 |
+
|
99 |
+
networks:
|
100 |
+
n8n-network:
|
101 |
+
driver: bridge
|
knowledge/.DS_Store
ADDED
Binary file (6.15 kB). View file
|
|
package.json
ADDED
@@ -0,0 +1,48 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
{
|
2 |
+
"name": "n8n-infrastructure",
|
3 |
+
"version": "1.0.0",
|
4 |
+
"description": "Production-ready n8n infrastructure with AI integrations and automated knowledge management",
|
5 |
+
"private": true,
|
6 |
+
"keywords": [
|
7 |
+
"n8n",
|
8 |
+
"workflow-automation",
|
9 |
+
"ai-integration",
|
10 |
+
"langchain",
|
11 |
+
"vector-database",
|
12 |
+
"hugging-face",
|
13 |
+
"supabase",
|
14 |
+
"devops"
|
15 |
+
],
|
16 |
+
"scripts": {
|
17 |
+
"dev": "docker compose -f docker/docker-compose.yml up --build",
|
18 |
+
"start": "docker compose -f docker/docker-compose.yml up -d",
|
19 |
+
"stop": "docker compose -f docker/docker-compose.yml down",
|
20 |
+
"logs": "docker compose -f docker/docker-compose.yml logs -f n8n",
|
21 |
+
"backup": "bash scripts/backup.sh",
|
22 |
+
"restore": "bash scripts/restore.sh",
|
23 |
+
"sync-knowledge": "bash scripts/sync-knowledge.sh",
|
24 |
+
"health-check": "curl -f http://localhost:7860/healthz",
|
25 |
+
"clean": "docker compose -f docker/docker-compose.yml down -v && docker system prune -f",
|
26 |
+
"deploy": "gh workflow run deploy-to-hf.yml",
|
27 |
+
"test": "bash scripts/test-infrastructure.sh"
|
28 |
+
},
|
29 |
+
"repository": {
|
30 |
+
"type": "git",
|
31 |
+
"url": "https://github.com/your-username/n8n-infra.git"
|
32 |
+
},
|
33 |
+
"author": {
|
34 |
+
"name": "Your Name",
|
35 |
+
"email": "[email protected]"
|
36 |
+
},
|
37 |
+
"license": "Apache-2.0",
|
38 |
+
"engines": {
|
39 |
+
"node": ">=18.0.0",
|
40 |
+
"npm": ">=9.0.0"
|
41 |
+
},
|
42 |
+
"devDependencies": {
|
43 |
+
"@types/node": "^20.0.0"
|
44 |
+
},
|
45 |
+
"dependencies": {
|
46 |
+
"huggingface_hub": "^0.20.1"
|
47 |
+
}
|
48 |
+
}
|
scripts/backup.sh
ADDED
@@ -0,0 +1,198 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
#!/bin/bash
|
2 |
+
|
3 |
+
# n8n Infrastructure Backup Script
|
4 |
+
# Backs up workflows, credentials, and configurations
|
5 |
+
# Usage: ./backup.sh [backup-name]
|
6 |
+
|
7 |
+
set -euo pipefail
|
8 |
+
|
9 |
+
# Configuration
|
10 |
+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
11 |
+
PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
|
12 |
+
BACKUP_DIR="$PROJECT_ROOT/workflows/backup"
|
13 |
+
TIMESTAMP=$(date +"%Y%m%d_%H%M%S")
|
14 |
+
BACKUP_NAME="${1:-n8n_backup_$TIMESTAMP}"
|
15 |
+
|
16 |
+
# Colors for output
|
17 |
+
RED='\033[0;31m'
|
18 |
+
GREEN='\033[0;32m'
|
19 |
+
YELLOW='\033[1;33m'
|
20 |
+
NC='\033[0m' # No Color
|
21 |
+
|
22 |
+
log_info() {
|
23 |
+
echo -e "${GREEN}[INFO]${NC} $1"
|
24 |
+
}
|
25 |
+
|
26 |
+
log_warn() {
|
27 |
+
echo -e "${YELLOW}[WARN]${NC} $1"
|
28 |
+
}
|
29 |
+
|
30 |
+
log_error() {
|
31 |
+
echo -e "${RED}[ERROR]${NC} $1"
|
32 |
+
}
|
33 |
+
|
34 |
+
# Check if Docker is running
|
35 |
+
check_docker() {
|
36 |
+
if ! docker ps > /dev/null 2>&1; then
|
37 |
+
log_error "Docker is not running or accessible"
|
38 |
+
exit 1
|
39 |
+
fi
|
40 |
+
}
|
41 |
+
|
42 |
+
# Create backup directory
|
43 |
+
create_backup_dir() {
|
44 |
+
local backup_path="$BACKUP_DIR/$BACKUP_NAME"
|
45 |
+
mkdir -p "$backup_path"
|
46 |
+
echo "$backup_path"
|
47 |
+
}
|
48 |
+
|
49 |
+
# Backup n8n workflows via API
|
50 |
+
backup_workflows() {
|
51 |
+
local backup_path="$1"
|
52 |
+
local container_name="n8n-automation"
|
53 |
+
|
54 |
+
log_info "Backing up n8n workflows..."
|
55 |
+
|
56 |
+
if docker ps --format '{{.Names}}' | grep -q "^$container_name$"; then
|
57 |
+
# Export workflows using n8n CLI inside container
|
58 |
+
docker exec "$container_name" n8n export:workflow --all --output="/home/node/.n8n/backup/workflows_$TIMESTAMP.json" || {
|
59 |
+
log_warn "Failed to export workflows via CLI, trying API approach"
|
60 |
+
return 1
|
61 |
+
}
|
62 |
+
|
63 |
+
# Copy exported file to backup directory
|
64 |
+
docker cp "$container_name:/home/node/.n8n/backup/workflows_$TIMESTAMP.json" "$backup_path/"
|
65 |
+
log_info "Workflows backed up successfully"
|
66 |
+
else
|
67 |
+
log_error "n8n container not found or not running"
|
68 |
+
return 1
|
69 |
+
fi
|
70 |
+
}
|
71 |
+
|
72 |
+
# Backup credentials (encrypted)
|
73 |
+
backup_credentials() {
|
74 |
+
local backup_path="$1"
|
75 |
+
local container_name="n8n-automation"
|
76 |
+
|
77 |
+
log_info "Backing up encrypted credentials..."
|
78 |
+
|
79 |
+
if docker ps --format '{{.Names}}' | grep -q "^$container_name$"; then
|
80 |
+
docker cp "$container_name:/home/node/.n8n/credentials" "$backup_path/" 2>/dev/null || {
|
81 |
+
log_warn "No credentials found or access denied"
|
82 |
+
}
|
83 |
+
fi
|
84 |
+
}
|
85 |
+
|
86 |
+
# Backup database schema and essential data
|
87 |
+
backup_database() {
|
88 |
+
local backup_path="$1"
|
89 |
+
|
90 |
+
log_info "Backing up database schema..."
|
91 |
+
|
92 |
+
if [[ -n "${DB_POSTGRESDB_PASSWORD:-}" ]]; then
|
93 |
+
export PGPASSWORD="$DB_POSTGRESDB_PASSWORD"
|
94 |
+
|
95 |
+
pg_dump \
|
96 |
+
--host="${DB_POSTGRESDB_HOST}" \
|
97 |
+
--port="${DB_POSTGRESDB_PORT:-5432}" \
|
98 |
+
--username="${DB_POSTGRESDB_USER}" \
|
99 |
+
--dbname="${DB_POSTGRESDB_DATABASE}" \
|
100 |
+
--schema-only \
|
101 |
+
--no-owner \
|
102 |
+
--no-privileges \
|
103 |
+
> "$backup_path/schema_$TIMESTAMP.sql" || {
|
104 |
+
log_error "Database backup failed"
|
105 |
+
return 1
|
106 |
+
}
|
107 |
+
|
108 |
+
log_info "Database schema backed up successfully"
|
109 |
+
else
|
110 |
+
log_warn "Database credentials not available, skipping database backup"
|
111 |
+
fi
|
112 |
+
}
|
113 |
+
|
114 |
+
# Backup knowledge base content
|
115 |
+
backup_knowledge() {
|
116 |
+
local backup_path="$1"
|
117 |
+
|
118 |
+
log_info "Backing up knowledge base..."
|
119 |
+
|
120 |
+
if [[ -d "$PROJECT_ROOT/knowledge" ]]; then
|
121 |
+
cp -r "$PROJECT_ROOT/knowledge" "$backup_path/"
|
122 |
+
log_info "Knowledge base backed up successfully"
|
123 |
+
else
|
124 |
+
log_warn "Knowledge base directory not found"
|
125 |
+
fi
|
126 |
+
}
|
127 |
+
|
128 |
+
# Create backup metadata
|
129 |
+
create_metadata() {
|
130 |
+
local backup_path="$1"
|
131 |
+
|
132 |
+
cat > "$backup_path/backup_metadata.json" << EOF
|
133 |
+
{
|
134 |
+
"backup_name": "$BACKUP_NAME",
|
135 |
+
"timestamp": "$TIMESTAMP",
|
136 |
+
"created_at": "$(date -Iseconds)",
|
137 |
+
"n8n_version": "$(docker exec n8n-automation n8n --version 2>/dev/null || echo 'unknown')",
|
138 |
+
"backup_type": "full",
|
139 |
+
"components": [
|
140 |
+
"workflows",
|
141 |
+
"credentials",
|
142 |
+
"database_schema",
|
143 |
+
"knowledge_base"
|
144 |
+
],
|
145 |
+
"notes": "Automated backup created by backup.sh script"
|
146 |
+
}
|
147 |
+
EOF
|
148 |
+
}
|
149 |
+
|
150 |
+
# Cleanup old backups
|
151 |
+
cleanup_old_backups() {
|
152 |
+
local retention_days="${BACKUP_RETENTION_DAYS:-30}"
|
153 |
+
|
154 |
+
log_info "Cleaning up backups older than $retention_days days..."
|
155 |
+
|
156 |
+
find "$BACKUP_DIR" -type d -name "n8n_backup_*" -mtime "+$retention_days" -exec rm -rf {} + 2>/dev/null || true
|
157 |
+
}
|
158 |
+
|
159 |
+
# Main backup process
|
160 |
+
main() {
|
161 |
+
log_info "Starting n8n infrastructure backup: $BACKUP_NAME"
|
162 |
+
|
163 |
+
# Preliminary checks
|
164 |
+
check_docker
|
165 |
+
|
166 |
+
# Load environment variables
|
167 |
+
if [[ -f "$PROJECT_ROOT/.env" ]]; then
|
168 |
+
source "$PROJECT_ROOT/.env"
|
169 |
+
fi
|
170 |
+
|
171 |
+
# Create backup directory
|
172 |
+
local backup_path=$(create_backup_dir)
|
173 |
+
log_info "Backup directory created: $backup_path"
|
174 |
+
|
175 |
+
# Perform backups
|
176 |
+
backup_workflows "$backup_path" || log_warn "Workflow backup incomplete"
|
177 |
+
backup_credentials "$backup_path" || log_warn "Credentials backup incomplete"
|
178 |
+
backup_database "$backup_path" || log_warn "Database backup incomplete"
|
179 |
+
backup_knowledge "$backup_path" || log_warn "Knowledge base backup incomplete"
|
180 |
+
|
181 |
+
# Create metadata
|
182 |
+
create_metadata "$backup_path"
|
183 |
+
|
184 |
+
# Cleanup old backups
|
185 |
+
cleanup_old_backups
|
186 |
+
|
187 |
+
log_info "Backup completed successfully: $backup_path"
|
188 |
+
|
189 |
+
# Optional: Create compressed archive
|
190 |
+
if command -v tar > /dev/null; then
|
191 |
+
local archive_name="$BACKUP_DIR/${BACKUP_NAME}.tar.gz"
|
192 |
+
tar -czf "$archive_name" -C "$BACKUP_DIR" "$BACKUP_NAME"
|
193 |
+
log_info "Compressed backup created: $archive_name"
|
194 |
+
fi
|
195 |
+
}
|
196 |
+
|
197 |
+
# Run main function
|
198 |
+
main "$@"
|
scripts/restore.sh
ADDED
@@ -0,0 +1,204 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
#!/bin/bash
|
2 |
+
|
3 |
+
# n8n Infrastructure Restore Script
|
4 |
+
# Restores workflows, credentials, and configurations from backup
|
5 |
+
# Usage: ./restore.sh <backup-name>
|
6 |
+
|
7 |
+
set -euo pipefail
|
8 |
+
|
9 |
+
# Configuration
|
10 |
+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
11 |
+
PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
|
12 |
+
BACKUP_DIR="$PROJECT_ROOT/workflows/backup"
|
13 |
+
|
14 |
+
# Colors for output
|
15 |
+
RED='\033[0;31m'
|
16 |
+
GREEN='\033[0;32m'
|
17 |
+
YELLOW='\033[1;33m'
|
18 |
+
NC='\033[0m' # No Color
|
19 |
+
|
20 |
+
log_info() {
|
21 |
+
echo -e "${GREEN}[INFO]${NC} $1"
|
22 |
+
}
|
23 |
+
|
24 |
+
log_warn() {
|
25 |
+
echo -e "${YELLOW}[WARN]${NC} $1"
|
26 |
+
}
|
27 |
+
|
28 |
+
log_error() {
|
29 |
+
echo -e "${RED}[ERROR]${NC} $1"
|
30 |
+
}
|
31 |
+
|
32 |
+
# Validate backup name argument
|
33 |
+
if [[ $# -eq 0 ]]; then
|
34 |
+
log_error "Usage: $0 <backup-name>"
|
35 |
+
log_info "Available backups:"
|
36 |
+
ls -1 "$BACKUP_DIR" | grep -E "^n8n_backup_" | head -10
|
37 |
+
exit 1
|
38 |
+
fi
|
39 |
+
|
40 |
+
BACKUP_NAME="$1"
|
41 |
+
BACKUP_PATH="$BACKUP_DIR/$BACKUP_NAME"
|
42 |
+
|
43 |
+
# Check if backup exists
|
44 |
+
if [[ ! -d "$BACKUP_PATH" ]]; then
|
45 |
+
# Try with .tar.gz extension
|
46 |
+
if [[ -f "$BACKUP_DIR/$BACKUP_NAME.tar.gz" ]]; then
|
47 |
+
log_info "Found compressed backup, extracting..."
|
48 |
+
tar -xzf "$BACKUP_DIR/$BACKUP_NAME.tar.gz" -C "$BACKUP_DIR"
|
49 |
+
else
|
50 |
+
log_error "Backup not found: $BACKUP_NAME"
|
51 |
+
exit 1
|
52 |
+
fi
|
53 |
+
fi
|
54 |
+
|
55 |
+
# Verify backup integrity
|
56 |
+
verify_backup() {
|
57 |
+
local metadata_file="$BACKUP_PATH/backup_metadata.json"
|
58 |
+
|
59 |
+
if [[ ! -f "$metadata_file" ]]; then
|
60 |
+
log_warn "Backup metadata not found, proceeding with caution"
|
61 |
+
return 0
|
62 |
+
fi
|
63 |
+
|
64 |
+
log_info "Backup verification:"
|
65 |
+
cat "$metadata_file" | jq -r '.backup_name, .created_at, .backup_type'
|
66 |
+
}
|
67 |
+
|
68 |
+
# Restore workflows
|
69 |
+
restore_workflows() {
|
70 |
+
local container_name="n8n-automation"
|
71 |
+
|
72 |
+
log_info "Restoring workflows..."
|
73 |
+
|
74 |
+
if ! docker ps --format '{{.Names}}' | grep -q "^$container_name$"; then
|
75 |
+
log_error "n8n container not running. Start the container first."
|
76 |
+
return 1
|
77 |
+
fi
|
78 |
+
|
79 |
+
# Find workflow backup file
|
80 |
+
local workflow_file=$(find "$BACKUP_PATH" -name "workflows_*.json" | head -1)
|
81 |
+
|
82 |
+
if [[ -n "$workflow_file" ]]; then
|
83 |
+
# Copy workflow file to container
|
84 |
+
docker cp "$workflow_file" "$container_name:/tmp/workflows_restore.json"
|
85 |
+
|
86 |
+
# Import workflows
|
87 |
+
docker exec "$container_name" n8n import:workflow --input="/tmp/workflows_restore.json" || {
|
88 |
+
log_error "Failed to import workflows"
|
89 |
+
return 1
|
90 |
+
}
|
91 |
+
|
92 |
+
log_info "Workflows restored successfully"
|
93 |
+
else
|
94 |
+
log_warn "No workflow backup file found"
|
95 |
+
fi
|
96 |
+
}
|
97 |
+
|
98 |
+
# Restore credentials
|
99 |
+
restore_credentials() {
|
100 |
+
local container_name="n8n-automation"
|
101 |
+
|
102 |
+
log_info "Restoring credentials..."
|
103 |
+
|
104 |
+
if [[ -d "$BACKUP_PATH/credentials" ]]; then
|
105 |
+
docker cp "$BACKUP_PATH/credentials/." "$container_name:/home/node/.n8n/credentials/"
|
106 |
+
log_info "Credentials restored successfully"
|
107 |
+
else
|
108 |
+
log_warn "No credentials backup found"
|
109 |
+
fi
|
110 |
+
}
|
111 |
+
|
112 |
+
# Restore database (schema only, data should be preserved)
|
113 |
+
restore_database() {
|
114 |
+
log_info "Restoring database schema..."
|
115 |
+
|
116 |
+
local schema_file=$(find "$BACKUP_PATH" -name "schema_*.sql" | head -1)
|
117 |
+
|
118 |
+
if [[ -n "$schema_file" && -n "${DB_POSTGRESDB_PASSWORD:-}" ]]; then
|
119 |
+
export PGPASSWORD="$DB_POSTGRESDB_PASSWORD"
|
120 |
+
|
121 |
+
log_warn "This will update database schema. Proceed? (y/N)"
|
122 |
+
read -r response
|
123 |
+
if [[ "$response" =~ ^[Yy]$ ]]; then
|
124 |
+
psql \
|
125 |
+
--host="${DB_POSTGRESDB_HOST}" \
|
126 |
+
--port="${DB_POSTGRESDB_PORT:-5432}" \
|
127 |
+
--username="${DB_POSTGRESDB_USER}" \
|
128 |
+
--dbname="${DB_POSTGRESDB_DATABASE}" \
|
129 |
+
--file="$schema_file" || {
|
130 |
+
log_error "Database restore failed"
|
131 |
+
return 1
|
132 |
+
}
|
133 |
+
log_info "Database schema restored successfully"
|
134 |
+
else
|
135 |
+
log_info "Database restore skipped"
|
136 |
+
fi
|
137 |
+
else
|
138 |
+
log_warn "No database backup found or credentials missing"
|
139 |
+
fi
|
140 |
+
}
|
141 |
+
|
142 |
+
# Restart services after restore
|
143 |
+
restart_services() {
|
144 |
+
log_info "Restarting n8n services..."
|
145 |
+
|
146 |
+
docker-compose -f "$PROJECT_ROOT/docker/docker-compose.yml" restart n8n
|
147 |
+
|
148 |
+
# Wait for service to be ready
|
149 |
+
local max_attempts=30
|
150 |
+
local attempt=1
|
151 |
+
|
152 |
+
while [[ $attempt -le $max_attempts ]]; do
|
153 |
+
if curl -f http://localhost:7860/healthz > /dev/null 2>&1; then
|
154 |
+
log_info "n8n service is ready"
|
155 |
+
break
|
156 |
+
fi
|
157 |
+
|
158 |
+
log_info "Waiting for n8n to start... ($attempt/$max_attempts)"
|
159 |
+
sleep 10
|
160 |
+
((attempt++))
|
161 |
+
done
|
162 |
+
|
163 |
+
if [[ $attempt -gt $max_attempts ]]; then
|
164 |
+
log_error "n8n failed to start after restore"
|
165 |
+
return 1
|
166 |
+
fi
|
167 |
+
}
|
168 |
+
|
169 |
+
# Main restore process
|
170 |
+
main() {
|
171 |
+
log_info "Starting n8n infrastructure restore: $BACKUP_NAME"
|
172 |
+
|
173 |
+
# Load environment variables
|
174 |
+
if [[ -f "$PROJECT_ROOT/.env" ]]; then
|
175 |
+
source "$PROJECT_ROOT/.env"
|
176 |
+
fi
|
177 |
+
|
178 |
+
# Verify backup
|
179 |
+
verify_backup
|
180 |
+
|
181 |
+
# Confirm restore operation
|
182 |
+
log_warn "This will restore n8n configuration from backup: $BACKUP_NAME"
|
183 |
+
log_warn "Current workflows and credentials may be overwritten. Continue? (y/N)"
|
184 |
+
read -r response
|
185 |
+
|
186 |
+
if [[ ! "$response" =~ ^[Yy]$ ]]; then
|
187 |
+
log_info "Restore operation cancelled"
|
188 |
+
exit 0
|
189 |
+
fi
|
190 |
+
|
191 |
+
# Perform restore operations
|
192 |
+
restore_workflows || log_warn "Workflow restore incomplete"
|
193 |
+
restore_credentials || log_warn "Credentials restore incomplete"
|
194 |
+
restore_database || log_warn "Database restore incomplete"
|
195 |
+
|
196 |
+
# Restart services
|
197 |
+
restart_services
|
198 |
+
|
199 |
+
log_info "Restore completed successfully"
|
200 |
+
log_info "Access your n8n instance at: ${WEBHOOK_URL:-http://localhost:7860}"
|
201 |
+
}
|
202 |
+
|
203 |
+
# Run main function
|
204 |
+
main "$@"
|
scripts/sync-knowledge.sh
ADDED
@@ -0,0 +1,367 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
#!/bin/bash
|
2 |
+
|
3 |
+
# Knowledge Base Synchronization Script
|
4 |
+
# Syncs content from multiple GitHub repositories and generates embeddings
|
5 |
+
# Usage: ./sync-knowledge.sh
|
6 |
+
|
7 |
+
set -euo pipefail
|
8 |
+
|
9 |
+
# Configuration
|
10 |
+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
11 |
+
PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
|
12 |
+
KNOWLEDGE_DIR="$PROJECT_ROOT/knowledge"
|
13 |
+
TEMP_DIR="/tmp/kb-sync-$$"
|
14 |
+
|
15 |
+
# Colors for output
|
16 |
+
RED='\033[0;31m'
|
17 |
+
GREEN='\033[0;32m'
|
18 |
+
YELLOW='\033[1;33m'
|
19 |
+
BLUE='\033[0;34m'
|
20 |
+
NC='\033[0m' # No Color
|
21 |
+
|
22 |
+
log_info() {
|
23 |
+
echo -e "${GREEN}[INFO]${NC} $1"
|
24 |
+
}
|
25 |
+
|
26 |
+
log_warn() {
|
27 |
+
echo -e "${YELLOW}[WARN]${NC} $1"
|
28 |
+
}
|
29 |
+
|
30 |
+
log_error() {
|
31 |
+
echo -e "${RED}[ERROR]${NC} $1"
|
32 |
+
}
|
33 |
+
|
34 |
+
log_debug() {
|
35 |
+
echo -e "${BLUE}[DEBUG]${NC} $1"
|
36 |
+
}
|
37 |
+
|
38 |
+
# Cleanup function
|
39 |
+
cleanup() {
|
40 |
+
log_info "Cleaning up temporary files..."
|
41 |
+
rm -rf "$TEMP_DIR"
|
42 |
+
}
|
43 |
+
|
44 |
+
# Set trap for cleanup
|
45 |
+
trap cleanup EXIT
|
46 |
+
|
47 |
+
# Check dependencies
|
48 |
+
check_dependencies() {
|
49 |
+
local deps=("git" "curl" "jq")
|
50 |
+
|
51 |
+
for dep in "${deps[@]}"; do
|
52 |
+
if ! command -v "$dep" > /dev/null; then
|
53 |
+
log_error "Required dependency not found: $dep"
|
54 |
+
exit 1
|
55 |
+
fi
|
56 |
+
done
|
57 |
+
}
|
58 |
+
|
59 |
+
# Load environment variables
|
60 |
+
load_env() {
|
61 |
+
if [[ -f "$PROJECT_ROOT/.env" ]]; then
|
62 |
+
source "$PROJECT_ROOT/.env"
|
63 |
+
else
|
64 |
+
log_error ".env file not found. Copy .env.example and configure it."
|
65 |
+
exit 1
|
66 |
+
fi
|
67 |
+
}
|
68 |
+
|
69 |
+
# Clone or update repository
|
70 |
+
sync_repository() {
|
71 |
+
local repo_url="$1"
|
72 |
+
local target_path="$2"
|
73 |
+
local branch="${3:-main}"
|
74 |
+
local subpath="$4"
|
75 |
+
|
76 |
+
log_info "Syncing repository: $repo_url"
|
77 |
+
log_debug "Target: $target_path, Branch: $branch, Subpath: $subpath"
|
78 |
+
|
79 |
+
local repo_name=$(basename "$repo_url" .git)
|
80 |
+
local temp_repo_path="$TEMP_DIR/$repo_name"
|
81 |
+
|
82 |
+
# Clone repository to temp directory
|
83 |
+
git clone --depth 1 --branch "$branch" "$repo_url" "$temp_repo_path" || {
|
84 |
+
log_error "Failed to clone repository: $repo_url"
|
85 |
+
return 1
|
86 |
+
}
|
87 |
+
|
88 |
+
# Copy specific subpath to target
|
89 |
+
local source_path="$temp_repo_path/$subpath"
|
90 |
+
if [[ -d "$source_path" ]]; then
|
91 |
+
mkdir -p "$(dirname "$target_path")"
|
92 |
+
cp -r "$source_path/." "$target_path/"
|
93 |
+
log_info "Successfully synced to: $target_path"
|
94 |
+
else
|
95 |
+
log_warn "Subpath not found: $subpath in $repo_url"
|
96 |
+
return 1
|
97 |
+
fi
|
98 |
+
}
|
99 |
+
|
100 |
+
# Generate embeddings for knowledge content
|
101 |
+
generate_embeddings() {
|
102 |
+
local knowledge_path="$1"
|
103 |
+
local collection_name="$2"
|
104 |
+
|
105 |
+
log_info "Generating embeddings for: $collection_name"
|
106 |
+
|
107 |
+
# Create Python script for embedding generation
|
108 |
+
cat > "$TEMP_DIR/generate_embeddings.py" << 'EOF'
|
109 |
+
import os
|
110 |
+
import json
|
111 |
+
import sys
|
112 |
+
from pathlib import Path
|
113 |
+
import hashlib
|
114 |
+
import requests
|
115 |
+
from sentence_transformers import SentenceTransformer
|
116 |
+
|
117 |
+
def load_model():
|
118 |
+
"""Load sentence transformer model"""
|
119 |
+
try:
|
120 |
+
model = SentenceTransformer('all-MiniLM-L6-v2')
|
121 |
+
return model
|
122 |
+
except Exception as e:
|
123 |
+
print(f"Error loading model: {e}")
|
124 |
+
return None
|
125 |
+
|
126 |
+
def process_text_files(knowledge_path, collection_name):
|
127 |
+
"""Process text files and generate embeddings"""
|
128 |
+
model = load_model()
|
129 |
+
if not model:
|
130 |
+
return False
|
131 |
+
|
132 |
+
embeddings_data = []
|
133 |
+
knowledge_path = Path(knowledge_path)
|
134 |
+
|
135 |
+
# Process markdown and text files
|
136 |
+
for file_path in knowledge_path.rglob("*.md"):
|
137 |
+
try:
|
138 |
+
with open(file_path, 'r', encoding='utf-8') as f:
|
139 |
+
content = f.read()
|
140 |
+
|
141 |
+
# Generate embedding
|
142 |
+
embedding = model.encode(content).tolist()
|
143 |
+
|
144 |
+
# Create document metadata
|
145 |
+
doc_id = hashlib.md5(str(file_path).encode()).hexdigest()
|
146 |
+
|
147 |
+
embeddings_data.append({
|
148 |
+
"id": doc_id,
|
149 |
+
"content": content,
|
150 |
+
"embedding": embedding,
|
151 |
+
"metadata": {
|
152 |
+
"file_path": str(file_path.relative_to(knowledge_path)),
|
153 |
+
"file_name": file_path.name,
|
154 |
+
"collection": collection_name,
|
155 |
+
"content_type": "markdown",
|
156 |
+
"size": len(content)
|
157 |
+
}
|
158 |
+
})
|
159 |
+
|
160 |
+
except Exception as e:
|
161 |
+
print(f"Error processing file {file_path}: {e}")
|
162 |
+
|
163 |
+
# Save embeddings to JSON file
|
164 |
+
output_file = knowledge_path / f"{collection_name}_embeddings.json"
|
165 |
+
with open(output_file, 'w', encoding='utf-8') as f:
|
166 |
+
json.dump(embeddings_data, f, indent=2, ensure_ascii=False)
|
167 |
+
|
168 |
+
print(f"Generated {len(embeddings_data)} embeddings for {collection_name}")
|
169 |
+
return True
|
170 |
+
|
171 |
+
if __name__ == "__main__":
|
172 |
+
if len(sys.argv) != 3:
|
173 |
+
print("Usage: python generate_embeddings.py <knowledge_path> <collection_name>")
|
174 |
+
sys.exit(1)
|
175 |
+
|
176 |
+
knowledge_path = sys.argv[1]
|
177 |
+
collection_name = sys.argv[2]
|
178 |
+
|
179 |
+
if process_text_files(knowledge_path, collection_name):
|
180 |
+
print("Embedding generation completed successfully")
|
181 |
+
else:
|
182 |
+
print("Embedding generation failed")
|
183 |
+
sys.exit(1)
|
184 |
+
EOF
|
185 |
+
|
186 |
+
# Run embedding generation
|
187 |
+
python3 "$TEMP_DIR/generate_embeddings.py" "$knowledge_path" "$collection_name" || {
|
188 |
+
log_error "Failed to generate embeddings for $collection_name"
|
189 |
+
return 1
|
190 |
+
}
|
191 |
+
}
|
192 |
+
|
193 |
+
# Upload embeddings to vector store
|
194 |
+
upload_embeddings() {
|
195 |
+
local embeddings_file="$1"
|
196 |
+
local collection_name="$2"
|
197 |
+
|
198 |
+
log_info "Uploading embeddings to vector store: $collection_name"
|
199 |
+
|
200 |
+
if [[ ! -f "$embeddings_file" ]]; then
|
201 |
+
log_error "Embeddings file not found: $embeddings_file"
|
202 |
+
return 1
|
203 |
+
fi
|
204 |
+
|
205 |
+
# Upload to ChromaDB
|
206 |
+
local chroma_url="http://${CHROMA_HOST:-localhost}:${CHROMA_PORT:-8000}"
|
207 |
+
|
208 |
+
curl -X POST "$chroma_url/api/v1/collections" \
|
209 |
+
-H "Content-Type: application/json" \
|
210 |
+
-H "Authorization: Bearer ${CHROMA_AUTH_TOKEN}" \
|
211 |
+
-d "{\"name\": \"$collection_name\"}" || true
|
212 |
+
|
213 |
+
# Process and upload embeddings in batches
|
214 |
+
python3 - << EOF
|
215 |
+
import json
|
216 |
+
import requests
|
217 |
+
import sys
|
218 |
+
from pathlib import Path
|
219 |
+
|
220 |
+
def upload_batch(embeddings_data, collection_name, chroma_url, auth_token):
|
221 |
+
"""Upload embeddings in batches to ChromaDB"""
|
222 |
+
batch_size = 100
|
223 |
+
total_docs = len(embeddings_data)
|
224 |
+
|
225 |
+
for i in range(0, total_docs, batch_size):
|
226 |
+
batch = embeddings_data[i:i+batch_size]
|
227 |
+
|
228 |
+
# Prepare batch data for ChromaDB
|
229 |
+
ids = [doc["id"] for doc in batch]
|
230 |
+
embeddings = [doc["embedding"] for doc in batch]
|
231 |
+
metadatas = [doc["metadata"] for doc in batch]
|
232 |
+
documents = [doc["content"] for doc in batch]
|
233 |
+
|
234 |
+
payload = {
|
235 |
+
"ids": ids,
|
236 |
+
"embeddings": embeddings,
|
237 |
+
"metadatas": metadatas,
|
238 |
+
"documents": documents
|
239 |
+
}
|
240 |
+
|
241 |
+
try:
|
242 |
+
response = requests.post(
|
243 |
+
f"{chroma_url}/api/v1/collections/{collection_name}/add",
|
244 |
+
json=payload,
|
245 |
+
headers={
|
246 |
+
"Content-Type": "application/json",
|
247 |
+
"Authorization": f"Bearer {auth_token}"
|
248 |
+
},
|
249 |
+
timeout=30
|
250 |
+
)
|
251 |
+
|
252 |
+
if response.status_code == 200:
|
253 |
+
print(f"Uploaded batch {i//batch_size + 1} ({len(batch)} documents)")
|
254 |
+
else:
|
255 |
+
print(f"Error uploading batch {i//batch_size + 1}: {response.status_code}")
|
256 |
+
print(f"Response: {response.text}")
|
257 |
+
|
258 |
+
except Exception as e:
|
259 |
+
print(f"Error uploading batch {i//batch_size + 1}: {e}")
|
260 |
+
continue
|
261 |
+
|
262 |
+
# Load and upload embeddings
|
263 |
+
embeddings_file = "$embeddings_file"
|
264 |
+
collection_name = "$collection_name"
|
265 |
+
chroma_url = "$chroma_url"
|
266 |
+
auth_token = "${CHROMA_AUTH_TOKEN:-}"
|
267 |
+
|
268 |
+
try:
|
269 |
+
with open(embeddings_file, 'r', encoding='utf-8') as f:
|
270 |
+
embeddings_data = json.load(f)
|
271 |
+
|
272 |
+
upload_batch(embeddings_data, collection_name, chroma_url, auth_token)
|
273 |
+
print(f"Successfully uploaded {len(embeddings_data)} embeddings to {collection_name}")
|
274 |
+
|
275 |
+
except Exception as e:
|
276 |
+
print(f"Error: {e}")
|
277 |
+
sys.exit(1)
|
278 |
+
EOF
|
279 |
+
}
|
280 |
+
|
281 |
+
# Sync all knowledge repositories
|
282 |
+
sync_all_repositories() {
|
283 |
+
log_info "Starting knowledge base synchronization..."
|
284 |
+
|
285 |
+
mkdir -p "$TEMP_DIR"
|
286 |
+
|
287 |
+
# Repository configurations
|
288 |
+
declare -A repos=(
|
289 |
+
["n8n"]="${KB_REPO_N8N:-}:${KB_PATH_N8N:-projects/n8n}"
|
290 |
+
["videos-e-animacoes"]="${KB_REPO_N8N:-}:${KB_PATH_VIDEOS:-projects/videos-e-animacoes}"
|
291 |
+
["midjourney-prompt"]="${KB_REPO_N8N:-}:${KB_PATH_MIDJOURNEY:-projects/midjorney-prompt}"
|
292 |
+
)
|
293 |
+
|
294 |
+
for collection in "${!repos[@]}"; do
|
295 |
+
local repo_config="${repos[$collection]}"
|
296 |
+
local repo_url=$(echo "$repo_config" | cut -d':' -f1)
|
297 |
+
local subpath=$(echo "$repo_config" | cut -d':' -f2)
|
298 |
+
local target_path="$KNOWLEDGE_DIR/$collection"
|
299 |
+
|
300 |
+
if [[ -n "$repo_url" ]]; then
|
301 |
+
log_info "Syncing collection: $collection"
|
302 |
+
|
303 |
+
# Sync repository
|
304 |
+
sync_repository "$repo_url" "$target_path" "${KB_BRANCH_N8N:-main}" "$subpath"
|
305 |
+
|
306 |
+
# Generate embeddings
|
307 |
+
generate_embeddings "$target_path" "$collection"
|
308 |
+
|
309 |
+
# Upload to vector store
|
310 |
+
local embeddings_file="$target_path/${collection}_embeddings.json"
|
311 |
+
if [[ -f "$embeddings_file" ]]; then
|
312 |
+
upload_embeddings "$embeddings_file" "$collection"
|
313 |
+
fi
|
314 |
+
|
315 |
+
else
|
316 |
+
log_warn "Repository URL not configured for collection: $collection"
|
317 |
+
fi
|
318 |
+
done
|
319 |
+
}
|
320 |
+
|
321 |
+
# Update n8n with new knowledge
|
322 |
+
update_n8n_knowledge() {
|
323 |
+
log_info "Notifying n8n of knowledge base updates..."
|
324 |
+
|
325 |
+
# Create a webhook trigger to refresh knowledge in n8n workflows
|
326 |
+
if [[ -n "${WEBHOOK_URL:-}" ]]; then
|
327 |
+
local webhook_endpoint="$WEBHOOK_URL/webhook/knowledge-sync"
|
328 |
+
|
329 |
+
curl -X POST "$webhook_endpoint" \
|
330 |
+
-H "Content-Type: application/json" \
|
331 |
+
-d "{\"event\": \"knowledge_updated\", \"timestamp\": \"$(date -Iseconds)\"}" \
|
332 |
+
> /dev/null 2>&1 || {
|
333 |
+
log_warn "Failed to notify n8n of knowledge updates"
|
334 |
+
}
|
335 |
+
fi
|
336 |
+
}
|
337 |
+
|
338 |
+
# Main synchronization process
|
339 |
+
main() {
|
340 |
+
log_info "Starting knowledge base synchronization"
|
341 |
+
|
342 |
+
# Preliminary checks
|
343 |
+
check_dependencies
|
344 |
+
load_env
|
345 |
+
|
346 |
+
# Create knowledge directories
|
347 |
+
mkdir -p "$KNOWLEDGE_DIR"/{n8n,videos-e-animacoes,midjourney-prompt}
|
348 |
+
|
349 |
+
# Sync all repositories
|
350 |
+
sync_all_repositories
|
351 |
+
|
352 |
+
# Update n8n
|
353 |
+
update_n8n_knowledge
|
354 |
+
|
355 |
+
log_info "Knowledge base synchronization completed"
|
356 |
+
|
357 |
+
# Generate summary
|
358 |
+
log_info "Synchronization Summary:"
|
359 |
+
find "$KNOWLEDGE_DIR" -name "*_embeddings.json" -exec basename {} \; | while read file; do
|
360 |
+
local collection=$(echo "$file" | sed 's/_embeddings.json//')
|
361 |
+
local count=$(jq '. | length' "$KNOWLEDGE_DIR/$collection/$file" 2>/dev/null || echo "0")
|
362 |
+
log_info " - $collection: $count documents"
|
363 |
+
done
|
364 |
+
}
|
365 |
+
|
366 |
+
# Run main function
|
367 |
+
main "$@"
|
scripts/test-infrastructure.sh
ADDED
@@ -0,0 +1,190 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
#!/bin/bash
|
2 |
+
|
3 |
+
# Infrastructure Testing Script
|
4 |
+
# Tests all components of the n8n infrastructure
|
5 |
+
# Usage: ./test-infrastructure.sh
|
6 |
+
|
7 |
+
set -euo pipefail
|
8 |
+
|
9 |
+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
10 |
+
PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
|
11 |
+
|
12 |
+
# Colors
|
13 |
+
RED='\033[0;31m'
|
14 |
+
GREEN='\033[0;32m'
|
15 |
+
YELLOW='\033[1;33m'
|
16 |
+
BLUE='\033[0;34m'
|
17 |
+
NC='\033[0m'
|
18 |
+
|
19 |
+
log_info() { echo -e "${GREEN}[INFO]${NC} $1"; }
|
20 |
+
log_warn() { echo -e "${YELLOW}[WARN]${NC} $1"; }
|
21 |
+
log_error() { echo -e "${RED}[ERROR]${NC} $1"; }
|
22 |
+
log_test() { echo -e "${BLUE}[TEST]${NC} $1"; }
|
23 |
+
|
24 |
+
# Test results tracking
|
25 |
+
TESTS_TOTAL=0
|
26 |
+
TESTS_PASSED=0
|
27 |
+
TESTS_FAILED=0
|
28 |
+
|
29 |
+
run_test() {
|
30 |
+
local test_name="$1"
|
31 |
+
local test_command="$2"
|
32 |
+
|
33 |
+
((TESTS_TOTAL++))
|
34 |
+
log_test "Running: $test_name"
|
35 |
+
|
36 |
+
if eval "$test_command" > /dev/null 2>&1; then
|
37 |
+
echo " ✅ PASSED"
|
38 |
+
((TESTS_PASSED++))
|
39 |
+
return 0
|
40 |
+
else
|
41 |
+
echo " ❌ FAILED"
|
42 |
+
((TESTS_FAILED++))
|
43 |
+
return 1
|
44 |
+
fi
|
45 |
+
}
|
46 |
+
|
47 |
+
# Load environment
|
48 |
+
if [[ -f "$PROJECT_ROOT/.env" ]]; then
|
49 |
+
source "$PROJECT_ROOT/.env"
|
50 |
+
fi
|
51 |
+
|
52 |
+
# Test Docker configuration
|
53 |
+
test_docker() {
|
54 |
+
log_info "Testing Docker configuration..."
|
55 |
+
|
56 |
+
run_test "Docker daemon accessible" "docker --version"
|
57 |
+
run_test "Docker Compose available" "docker-compose --version"
|
58 |
+
run_test "Dockerfile syntax" "docker build -f docker/Dockerfile --dry-run . 2>/dev/null || docker build -f docker/Dockerfile -t n8n-test . --dry-run"
|
59 |
+
run_test "Docker Compose syntax" "docker-compose -f docker/docker-compose.yml config"
|
60 |
+
}
|
61 |
+
|
62 |
+
# Test environment configuration
|
63 |
+
test_environment() {
|
64 |
+
log_info "Testing environment configuration..."
|
65 |
+
|
66 |
+
run_test "Environment example exists" "[[ -f config/.env.example ]]"
|
67 |
+
run_test "Required directories exist" "[[ -d workflows && -d knowledge && -d scripts ]]"
|
68 |
+
|
69 |
+
if [[ -f .env ]]; then
|
70 |
+
run_test "Environment file loaded" "[[ -n \${N8N_ENCRYPTION_KEY:-} ]]"
|
71 |
+
run_test "Database config present" "[[ -n \${DB_POSTGRESDB_HOST:-} ]]"
|
72 |
+
else
|
73 |
+
log_warn "No .env file found - using example for testing"
|
74 |
+
fi
|
75 |
+
}
|
76 |
+
|
77 |
+
# Test scripts
|
78 |
+
test_scripts() {
|
79 |
+
log_info "Testing infrastructure scripts..."
|
80 |
+
|
81 |
+
run_test "Backup script executable" "[[ -x scripts/backup.sh ]]"
|
82 |
+
run_test "Restore script executable" "[[ -x scripts/restore.sh ]]"
|
83 |
+
run_test "Sync script executable" "[[ -x scripts/sync-knowledge.sh ]]"
|
84 |
+
run_test "Backup script syntax" "bash -n scripts/backup.sh"
|
85 |
+
run_test "Restore script syntax" "bash -n scripts/restore.sh"
|
86 |
+
run_test "Sync script syntax" "bash -n scripts/sync-knowledge.sh"
|
87 |
+
}
|
88 |
+
|
89 |
+
# Test GitHub Actions
|
90 |
+
test_github_actions() {
|
91 |
+
log_info "Testing GitHub Actions workflows..."
|
92 |
+
|
93 |
+
run_test "Deploy workflow exists" "[[ -f .github/workflows/deploy-to-hf.yml ]]"
|
94 |
+
run_test "Backup workflow exists" "[[ -f .github/workflows/backup-workflows.yml ]]"
|
95 |
+
run_test "Sync workflow exists" "[[ -f .github/workflows/sync-knowledge.yml ]]"
|
96 |
+
|
97 |
+
# Validate workflow syntax (basic YAML check)
|
98 |
+
if command -v python3 > /dev/null; then
|
99 |
+
run_test "Deploy workflow syntax" "python3 -c \"import yaml; yaml.safe_load(open('.github/workflows/deploy-to-hf.yml'))\""
|
100 |
+
run_test "Backup workflow syntax" "python3 -c \"import yaml; yaml.safe_load(open('.github/workflows/backup-workflows.yml'))\""
|
101 |
+
run_test "Sync workflow syntax" "python3 -c \"import yaml; yaml.safe_load(open('.github/workflows/sync-knowledge.yml'))\""
|
102 |
+
fi
|
103 |
+
}
|
104 |
+
|
105 |
+
# Test database connectivity
|
106 |
+
test_database() {
|
107 |
+
log_info "Testing database connectivity..."
|
108 |
+
|
109 |
+
if [[ -n "${DB_POSTGRESDB_HOST:-}" && -n "${DB_POSTGRESDB_PASSWORD:-}" ]]; then
|
110 |
+
export PGPASSWORD="$DB_POSTGRESDB_PASSWORD"
|
111 |
+
|
112 |
+
run_test "Database connection" "pg_isready -h '$DB_POSTGRESDB_HOST' -p '${DB_POSTGRESDB_PORT:-5432}' -U '$DB_POSTGRESDB_USER'"
|
113 |
+
run_test "Database access" "psql -h '$DB_POSTGRESDB_HOST' -p '${DB_POSTGRESDB_PORT:-5432}' -U '$DB_POSTGRESDB_USER' -d '$DB_POSTGRESDB_DATABASE' -c 'SELECT 1;'"
|
114 |
+
run_test "pgvector extension" "psql -h '$DB_POSTGRESDB_HOST' -p '${DB_POSTGRESDB_PORT:-5432}' -U '$DB_POSTGRESDB_USER' -d '$DB_POSTGRESDB_DATABASE' -c 'SELECT * FROM pg_extension WHERE extname = \\'vector\\';'"
|
115 |
+
else
|
116 |
+
log_warn "Database credentials not available - skipping connectivity tests"
|
117 |
+
fi
|
118 |
+
}
|
119 |
+
|
120 |
+
# Test knowledge base
|
121 |
+
test_knowledge() {
|
122 |
+
log_info "Testing knowledge base configuration..."
|
123 |
+
|
124 |
+
run_test "Knowledge directories exist" "[[ -d knowledge/n8n && -d knowledge/videos-e-animacoes && -d knowledge/midjourney-prompt ]]"
|
125 |
+
|
126 |
+
# Test Python dependencies for embedding generation
|
127 |
+
if command -v python3 > /dev/null; then
|
128 |
+
run_test "Python available" "python3 --version"
|
129 |
+
run_test "Required Python packages" "python3 -c 'import sentence_transformers, json, hashlib'"
|
130 |
+
fi
|
131 |
+
}
|
132 |
+
|
133 |
+
# Integration tests
|
134 |
+
test_integration() {
|
135 |
+
log_info "Running integration tests..."
|
136 |
+
|
137 |
+
# Test if we can start services locally
|
138 |
+
if run_test "Start services locally" "docker-compose -f docker/docker-compose.yml up -d --build"; then
|
139 |
+
sleep 30
|
140 |
+
|
141 |
+
run_test "n8n health endpoint" "curl -f http://localhost:7860/healthz"
|
142 |
+
run_test "Vector store endpoint" "curl -f http://localhost:8000/api/v1/heartbeat"
|
143 |
+
|
144 |
+
# Cleanup
|
145 |
+
docker-compose -f docker/docker-compose.yml down > /dev/null 2>&1
|
146 |
+
else
|
147 |
+
log_error "Failed to start services - skipping integration tests"
|
148 |
+
fi
|
149 |
+
}
|
150 |
+
|
151 |
+
# Main test execution
|
152 |
+
main() {
|
153 |
+
log_info "🧪 Starting n8n Infrastructure Test Suite"
|
154 |
+
echo "================================================"
|
155 |
+
|
156 |
+
cd "$PROJECT_ROOT"
|
157 |
+
|
158 |
+
# Run all test categories
|
159 |
+
test_docker
|
160 |
+
test_environment
|
161 |
+
test_scripts
|
162 |
+
test_github_actions
|
163 |
+
test_database
|
164 |
+
test_knowledge
|
165 |
+
|
166 |
+
# Skip integration tests if Docker unavailable
|
167 |
+
if docker --version > /dev/null 2>&1; then
|
168 |
+
test_integration
|
169 |
+
else
|
170 |
+
log_warn "Docker not available - skipping integration tests"
|
171 |
+
fi
|
172 |
+
|
173 |
+
# Test summary
|
174 |
+
echo ""
|
175 |
+
echo "================================================"
|
176 |
+
log_info "🎯 Test Results Summary"
|
177 |
+
echo " Total Tests: $TESTS_TOTAL"
|
178 |
+
echo " Passed: $TESTS_PASSED"
|
179 |
+
echo " Failed: $TESTS_FAILED"
|
180 |
+
|
181 |
+
if [[ $TESTS_FAILED -eq 0 ]]; then
|
182 |
+
echo " Status: ✅ ALL TESTS PASSED"
|
183 |
+
exit 0
|
184 |
+
else
|
185 |
+
echo " Status: ❌ SOME TESTS FAILED"
|
186 |
+
exit 1
|
187 |
+
fi
|
188 |
+
}
|
189 |
+
|
190 |
+
main "$@"
|