Spaces:
danilonovais
/
Running on CPU Upgrade

danilonovais commited on
Commit
89f19e4
·
1 Parent(s): da48ebe

Initial commit: n8n infrastructure with AI and automation

Browse files

Add production-ready n8n infrastructure for Hugging Face Spaces, including Docker setup, environment configuration, AI integrations (LangChain, OpenAI, Anthropic, Google Vertex AI), vector store (ChromaDB), Supabase PostgreSQL support, automated backup and restore scripts, and knowledge base synchronization. Includes documentation, security policy, and example configuration files.

.DS_Store ADDED
Binary file (6.15 kB). View file
 
LICENSE ADDED
@@ -0,0 +1,171 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Apache License
2
+ Version 2.0, January 2004
3
+ http://www.apache.org/licenses/
4
+
5
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
6
+
7
+ 1. Definitions.
8
+
9
+ "License\" shall mean the terms and conditions for use, reproduction,
10
+ and distribution as defined by Sections 1 through 9 of this document.
11
+
12
+ "Licensor\" shall mean the copyright owner or entity granting the License.
13
+
14
+ "Legal Entity\" shall mean the union of the acting entity and all
15
+ other entities that control, are controlled by, or are under common
16
+ control with that entity. For the purposes of this definition,
17
+ "control\" means (i) the power, direct or indirect, to cause the
18
+ direction or management of such entity, whether by contract or
19
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
20
+ outstanding shares, or (iii) beneficial ownership of such entity.
21
+
22
+ "You" (or "Your") shall mean an individual or Legal Entity
23
+ exercising permissions granted by this License.
24
+
25
+ "Source\" form shall mean the preferred form for making modifications,
26
+ including but not limited to software source code, documentation
27
+ source, and configuration files.
28
+
29
+ "Object\" form shall mean any form resulting from mechanical
30
+ transformation or translation of a Source form, including but
31
+ not limited to compiled object code, generated documentation,
32
+ and conversions to other media types.
33
+
34
+ "Work\" shall mean the work of authorship, whether in Source or
35
+ Object form, made available under the License, as indicated by a
36
+ copyright notice that is included in or attached to the work
37
+ (which shall not include communications that are clearly marked or
38
+ otherwise designated in writing by the copyright owner as "Not a Work").
39
+
40
+ "Derivative Works\" shall mean any work, whether in Source or Object
41
+ form, that is based upon (or derived from) the Work and for which the
42
+ editorial revisions, annotations, elaborations, or other modifications
43
+ represent, as a whole, an original work of authorship. For the purposes
44
+ of this License, Derivative Works shall not include works that remain
45
+ separable from, or merely link (or bind by name) to the interfaces of,
46
+ the Work and derivative works thereof.
47
+
48
+ "Contribution\" shall mean any work of authorship, including
49
+ the original version of the Work and any modifications or additions
50
+ to that Work or Derivative Works thereof, that is intentionally
51
+ submitted to Licensor for inclusion in the Work by the copyright owner
52
+ or by an individual or Legal Entity authorized to submit on behalf of
53
+ the copyright owner. For the purposes of this definition, "submitted"
54
+ means any form of electronic, verbal, or written communication sent
55
+ to the Licensor or its representatives, including but not limited to
56
+ communication on electronic mailing lists, source code control
57
+ systems, and issue tracking systems that are managed by, or on behalf
58
+ of, the Licensor for the purpose of discussing and improving the Work,
59
+ but excluding communication that is conspicuously marked or otherwise
60
+ designated in writing by the copyright owner as "Not a Contribution."
61
+
62
+ 2. Grant of Copyright License. Subject to the terms and conditions of
63
+ this License, each Contributor hereby grants to You a perpetual,
64
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
65
+ copyright license to use, reproduce, modify, distribute, and prepare
66
+ Derivative Works of, and to publicly perform and publicly display the
67
+ Work and such Derivative Works in Source or Object form.
68
+
69
+ 3. Grant of Patent License. Subject to the terms and conditions of
70
+ this License, each Contributor hereby grants to You a perpetual,
71
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
72
+ (except as stated in this section) patent license to make, have made,
73
+ use, offer to sell, sell, import, and otherwise transfer the Work,
74
+ where such license applies only to those patent claims licensable
75
+ by such Contributor that are necessarily infringed by their
76
+ Contribution(s) alone or by combination of their Contribution(s)
77
+ with the Work to which such Contribution(s) was submitted. If You
78
+ institute patent litigation against any entity (including a
79
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
80
+ or a Contribution incorporated within the Work constitutes direct
81
+ or contributory patent infringement, then any patent licenses
82
+ granted to You under this License for that Work shall terminate
83
+ as of the date such litigation is filed.
84
+
85
+ 4. Redistribution. You may reproduce and distribute copies of the
86
+ Work or Derivative Works thereof in any medium, with or without
87
+ modifications, and in Source or Object form, provided that You
88
+ meet the following conditions:
89
+
90
+ (a) You must give any other recipients of the Work or
91
+ Derivative Works a copy of this License; and
92
+
93
+ (b) You must cause any modified files to carry prominent notices
94
+ stating that You changed the files; and
95
+
96
+ (c) You must retain, in the Source form of any Derivative Works
97
+ that You distribute, all copyright, trademark, patent, attribution
98
+ notices from the Source form of the Work, excluding those notices
99
+ that do not pertain to any part of the Derivative Works; and
100
+
101
+ (d) If the Work includes a "NOTICE\" text file as part of its
102
+ distribution, then any Derivative Works that You distribute must
103
+ include a readable copy of the attribution notices contained
104
+ within such NOTICE file, excluding those notices that do not
105
+ pertain to any part of the Derivative Works, in at least one
106
+ of the following places: within a NOTICE text file distributed
107
+ as part of the Derivative Works; within the Source form or
108
+ documentation, if provided along with the Derivative Works; or,
109
+ within a display generated by the Derivative Works, if and
110
+ wherever such third-party notices normally appear. The contents
111
+ of the NOTICE file are for informational purposes only and
112
+ do not modify the License. You may add Your own attribution
113
+ notices within Derivative Works that You distribute, alongside
114
+ or as an addendum to the NOTICE text from the Work, provided
115
+ that such additional attribution notices cannot be construed
116
+ as modifying the License.
117
+
118
+ You may add Your own copyright notice to Your modifications and
119
+ may provide additional or different license terms and conditions
120
+ for use, reproduction, or distribution of Your modifications, or
121
+ for any such Derivative Works as a whole, provided Your use,
122
+ reproduction, and distribution of the Work otherwise complies with
123
+ the conditions stated in this License.
124
+
125
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
126
+ any Contribution intentionally submitted for inclusion in the Work
127
+ by You to the Licensor shall be under the terms and conditions of
128
+ this License, without any additional terms or conditions.
129
+ Notwithstanding the above, nothing herein shall supersede or modify
130
+ the terms of any separate license agreement you may have executed
131
+ with Licensor regarding such Contributions.
132
+
133
+ 6. Trademarks. This License does not grant permission to use the trade
134
+ names, trademarks, service marks, or product names of the Licensor,
135
+ except as required for reasonable and customary use in describing the
136
+ origin of the Work and reproducing the content of the NOTICE file.
137
+
138
+ 7. Disclaimer of Warranty. Unless required by applicable law or
139
+ agreed to in writing, Licensor provides the Work (and each
140
+ Contributor provides its Contributions) on an "AS IS\" BASIS,
141
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
142
+ implied, including, without limitation, any warranties or conditions
143
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
144
+ PARTICULAR PURPOSE. You are solely responsible for determining the
145
+ appropriateness of using or redistributing the Work and assume any
146
+ risks associated with Your exercise of permissions under this License.
147
+
148
+ 8. Limitation of Liability. In no event and under no legal theory,
149
+ whether in tort (including negligence), contract, or otherwise,
150
+ unless required by applicable law (such as deliberate and grossly
151
+ negligent acts) or agreed to in writing, shall any Contributor be
152
+ liable to You for damages, including any direct, indirect, special,
153
+ incidental, or consequential damages of any character arising as a
154
+ result of this License or out of the use or inability to use the
155
+ Work (including but not limited to damages for loss of goodwill,
156
+ work stoppage, computer failure or malfunction, or any and all
157
+ other commercial damages or losses), even if such Contributor
158
+ has been advised of the possibility of such damages.
159
+
160
+ 9. Accepting Warranty or Additional Liability. When redistributing
161
+ the Work or Derivative Works thereof, You may choose to offer,
162
+ and charge a fee for, acceptance of support, warranty, indemnity,
163
+ or other liability obligations and/or rights consistent with this
164
+ License. However, in accepting such obligations, You may act only
165
+ on Your own behalf and on Your sole responsibility, not on behalf
166
+ of any other Contributor, and only if You agree to indemnify,
167
+ defend, and hold each Contributor harmless for any liability
168
+ incurred by, or claims asserted against, such Contributor by reason
169
+ of your accepting any such warranty or additional liability.
170
+
171
+ END OF TERMS AND CONDITIONS
README.md ADDED
@@ -0,0 +1,458 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # n8n Infrastructure Repository
2
+
3
+ A comprehensive, production-ready infrastructure setup for deploying n8n automation platform on Hugging Face Spaces with AI integrations and automated knowledge management.
4
+
5
+ ## 🚀 Features
6
+
7
+ ### Core Platform
8
+ - **n8n v1.17.1**: Self-hosted workflow automation platform
9
+ - **Hugging Face Spaces**: Docker-based deployment with automatic scaling
10
+ - **Supabase PostgreSQL**: SSL-encrypted database with pgvector extension
11
+ - **ChromaDB**: Vector store for embeddings and AI-powered search
12
+
13
+ ### AI & Automation
14
+ - **LangChain Integration**: Advanced AI workflow capabilities
15
+ - **Multi-Model Support**: OpenAI GPT, Anthropic Claude, Google Vertex AI
16
+ - **Vector Knowledge Base**: Automated content ingestion with embeddings
17
+ - **Community Nodes**: Extended functionality with custom AI nodes
18
+
19
+ ### DevOps & Monitoring
20
+ - **GitHub Actions CI/CD**: Automated deployment and maintenance
21
+ - **Automated Backups**: Daily workflow and configuration backups
22
+ - **Knowledge Sync**: Multi-repository content synchronization
23
+ - **Health Monitoring**: Container health checks and alerting
24
+
25
+ ## 📋 Prerequisites
26
+
27
+ Before setting up the infrastructure, ensure you have:
28
+
29
+ 1. **GitHub Account** with repository access
30
+ 2. **Hugging Face Account** with Spaces access
31
+ 3. **Supabase Account** with PostgreSQL database
32
+ 4. **Git** and **Docker** installed locally
33
+
34
+ ### Required Secrets
35
+
36
+ Configure these secrets in your GitHub repository settings:
37
+
38
+ ```bash
39
+ # Hugging Face
40
+ HF_USERNAME=your-huggingface-username
41
+ HF_TOKEN=your-hf-token
42
+ HF_SPACE_NAME=n8n-automation
43
+
44
+ # Database
45
+ DB_POSTGRESDB_HOST=your-project.supabase.co
46
+ DB_POSTGRESDB_USER=postgres
47
+ DB_POSTGRESDB_PASSWORD=your-database-password
48
+ DB_POSTGRESDB_DATABASE=postgres
49
+
50
+ # n8n Configuration
51
+ N8N_ENCRYPTION_KEY=your-32-character-encryption-key
52
+ N8N_USER_MANAGEMENT_JWT_SECRET=your-jwt-secret
53
+ WEBHOOK_URL=https://your-username-n8n-automation.hf.space
54
+
55
+ # AI Services (Optional)
56
+ OPENAI_API_KEY=sk-your-openai-key
57
+ ANTHROPIC_API_KEY=sk-ant-your-anthropic-key
58
+ GOOGLE_PROJECT_ID=your-gcp-project
59
+ ```
60
+
61
+ ## 🛠️ Quick Start
62
+
63
+ ### 1. Repository Setup
64
+
65
+ ```bash
66
+ # Clone the repository
67
+ git clone https://github.com/your-username/n8n-infra.git
68
+ cd n8n-infra
69
+
70
+ # Create environment configuration
71
+ cp config/.env.example .env
72
+ # Edit .env with your actual values
73
+ ```
74
+
75
+ ### 2. Local Development
76
+
77
+ ```bash
78
+ # Start the full stack locally
79
+ docker-compose -f docker/docker-compose.yml up -d
80
+
81
+ # Check service status
82
+ docker-compose -f docker/docker-compose.yml ps
83
+
84
+ # View logs
85
+ docker-compose -f docker/docker-compose.yml logs -f n8n
86
+ ```
87
+
88
+ ### 3. Hugging Face Deployment
89
+
90
+ ```bash
91
+ # Trigger deployment via GitHub Actions
92
+ git push origin main
93
+
94
+ # Or deploy manually
95
+ gh workflow run deploy-to-hf.yml
96
+ ```
97
+
98
+ ## 📊 Database Setup
99
+
100
+ ### Supabase Configuration
101
+
102
+ 1. **Create Supabase Project**:
103
+ ```sql
104
+ -- Enable pgvector extension
105
+ CREATE EXTENSION IF NOT EXISTS vector;
106
+
107
+ -- Create knowledge base schema
108
+ CREATE SCHEMA IF NOT EXISTS knowledge;
109
+
110
+ -- Create embeddings table
111
+ CREATE TABLE knowledge.embeddings (
112
+ id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
113
+ content_id TEXT NOT NULL,
114
+ collection_name TEXT NOT NULL,
115
+ content TEXT NOT NULL,
116
+ embedding VECTOR(384),
117
+ metadata JSONB DEFAULT '{}',
118
+ created_at TIMESTAMPTZ DEFAULT NOW(),
119
+ updated_at TIMESTAMPTZ DEFAULT NOW()
120
+ );
121
+
122
+ -- Create indexes for performance
123
+ CREATE INDEX IF NOT EXISTS idx_embeddings_collection ON knowledge.embeddings(collection_name);
124
+ CREATE INDEX IF NOT EXISTS idx_embeddings_content_id ON knowledge.embeddings(content_id);
125
+ CREATE INDEX IF NOT EXISTS idx_embeddings_vector ON knowledge.embeddings
126
+ USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100);
127
+ ```
128
+
129
+ 2. **Configure Row Level Security**:
130
+ ```sql
131
+ -- Enable RLS
132
+ ALTER TABLE knowledge.embeddings ENABLE ROW LEVEL SECURITY;
133
+
134
+ -- Allow authenticated users to read embeddings
135
+ CREATE POLICY "Users can read embeddings" ON knowledge.embeddings
136
+ FOR SELECT TO authenticated USING (true);
137
+
138
+ -- Allow service role to manage embeddings
139
+ CREATE POLICY "Service role can manage embeddings" ON knowledge.embeddings
140
+ FOR ALL TO service_role USING (true);
141
+ ```
142
+
143
+ ## 🤖 AI Integration Guide
144
+
145
+ ### LangChain Workflows
146
+
147
+ The platform supports advanced LangChain workflows:
148
+
149
+ ```javascript
150
+ // Example: Knowledge-based Q&A workflow
151
+ {
152
+ "nodes": [
153
+ {
154
+ "name": "Vector Search",
155
+ "type": "n8n-nodes-vector-store",
156
+ "parameters": {
157
+ "operation": "similarity_search",
158
+ "query": "{{ $json.question }}",
159
+ "collection": "n8n",
160
+ "top_k": 5
161
+ }
162
+ },
163
+ {
164
+ "name": "LangChain QA",
165
+ "type": "@n8n/n8n-nodes-langchain",
166
+ "parameters": {
167
+ "chain_type": "question_answering",
168
+ "context": "{{ $json.vector_results }}",
169
+ "question": "{{ $json.question }}"
170
+ }
171
+ }
172
+ ]
173
+ }
174
+ ```
175
+
176
+ ### Custom AI Nodes
177
+
178
+ Install additional AI nodes:
179
+
180
+ ```bash
181
+ # Install in running container
182
+ docker exec n8n-automation npm install n8n-nodes-google-vertex-ai
183
+ docker exec n8n-automation npm install n8n-nodes-openai-advanced
184
+
185
+ # Restart to load new nodes
186
+ docker-compose -f docker/docker-compose.yml restart n8n
187
+ ```
188
+
189
+ ## 🗄️ Knowledge Management
190
+
191
+ ### Automated Synchronization
192
+
193
+ The system automatically syncs content from these repositories:
194
+
195
+ - **n8n Knowledge**: `/projects/n8n` - Workflow examples and best practices
196
+ - **Video & Animation**: `/projects/videos-e-animacoes` - Multimedia processing guides
197
+ - **Midjourney Prompts**: `/projects/midjorney-prompt` - AI art generation prompts
198
+
199
+ ### Manual Knowledge Sync
200
+
201
+ ```bash
202
+ # Sync specific collection
203
+ ./scripts/sync-knowledge.sh
204
+
205
+ # Or trigger via GitHub Actions
206
+ gh workflow run sync-knowledge.yml -f collections=n8n,midjourney-prompt
207
+ ```
208
+
209
+ ### Vector Search Setup
210
+
211
+ Query the knowledge base in n8n workflows:
212
+
213
+ ```javascript
214
+ // Vector similarity search node configuration
215
+ {
216
+ "collection": "n8n",
217
+ "query": "How to create webhook workflows",
218
+ "top_k": 3,
219
+ "score_threshold": 0.7
220
+ }
221
+ ```
222
+
223
+ ## 💾 Backup & Recovery
224
+
225
+ ### Automated Backups
226
+
227
+ Daily backups include:
228
+ - All n8n workflows (exported as JSON)
229
+ - Encrypted credentials
230
+ - Database schema
231
+ - Knowledge base content
232
+ - Vector embeddings
233
+
234
+ ### Manual Backup
235
+
236
+ ```bash
237
+ # Create full backup
238
+ ./scripts/backup.sh custom-backup-name
239
+
240
+ # List available backups
241
+ ls workflows/backup/
242
+
243
+ # Restore from backup
244
+ ./scripts/restore.sh n8n_backup_20240115_140230
245
+ ```
246
+
247
+ ### Backup Schedule
248
+
249
+ - **Daily**: Automated workflow backup at 2 AM UTC
250
+ - **Weekly**: Full system backup including database
251
+ - **On-demand**: Manual backups via GitHub Actions
252
+
253
+ ## 🔧 Maintenance
254
+
255
+ ### Health Monitoring
256
+
257
+ ```bash
258
+ # Check container health
259
+ docker-compose -f docker/docker-compose.yml ps
260
+
261
+ # View application logs
262
+ docker-compose -f docker/docker-compose.yml logs -f n8n
263
+
264
+ # Monitor vector store
265
+ curl http://localhost:8000/api/v1/heartbeat
266
+ ```
267
+
268
+ ### Performance Tuning
269
+
270
+ **Database Optimization**:
271
+ ```sql
272
+ -- Monitor query performance
273
+ SELECT query, mean_exec_time, calls
274
+ FROM pg_stat_statements
275
+ WHERE query LIKE '%n8n%'
276
+ ORDER BY mean_exec_time DESC
277
+ LIMIT 10;
278
+
279
+ -- Optimize vector searches
280
+ SET ivfflat.probes = 10;
281
+ ```
282
+
283
+ **Container Resources**:
284
+ ```yaml
285
+ # docker-compose.yml resource limits
286
+ services:
287
+ n8n:
288
+ deploy:
289
+ resources:
290
+ limits:
291
+ cpus: '2.0'
292
+ memory: 4G
293
+ reservations:
294
+ cpus: '1.0'
295
+ memory: 2G
296
+ ```
297
+
298
+ ## 🔒 Security
299
+
300
+ ### SSL Configuration
301
+
302
+ - All database connections use SSL encryption
303
+ - Webhook URLs must use HTTPS
304
+ - Container communication over encrypted networks
305
+
306
+ ### Credential Management
307
+
308
+ ```bash
309
+ # Credentials are encrypted by n8n
310
+ # Store sensitive files in config/credentials/
311
+ mkdir -p config/credentials
312
+ echo '{}' > config/credentials/google-service-account.json
313
+
314
+ # Set proper permissions
315
+ chmod 600 config/credentials/*
316
+ ```
317
+
318
+ ### Environment Security
319
+
320
+ - Never commit `.env` files
321
+ - Use GitHub Secrets for sensitive data
322
+ - Rotate encryption keys regularly
323
+ - Enable Supabase RLS policies
324
+
325
+ ## 🚨 Troubleshooting
326
+
327
+ ### Common Issues
328
+
329
+ **Connection Problems**:
330
+ ```bash
331
+ # Test database connection
332
+ docker exec n8n-automation psql "$DB_POSTGRESDB_HOST" -U "$DB_POSTGRESDB_USER" -c "\l"
333
+
334
+ # Check n8n logs
335
+ docker logs n8n-automation --tail 50
336
+
337
+ # Verify webhook connectivity
338
+ curl -I "$WEBHOOK_URL/healthz"
339
+ ```
340
+
341
+ **Deployment Issues**:
342
+ ```bash
343
+ # Check Hugging Face Space status
344
+ curl -I "https://huggingface.co/spaces/$HF_USERNAME/$HF_SPACE_NAME"
345
+
346
+ # View GitHub Actions logs
347
+ gh run list --workflow=deploy-to-hf.yml
348
+ gh run view [run-id] --log
349
+ ```
350
+
351
+ **Knowledge Sync Problems**:
352
+ ```bash
353
+ # Manual knowledge sync debug
354
+ ./scripts/sync-knowledge.sh
355
+ echo $? # Should return 0 for success
356
+
357
+ # Check embedding generation
358
+ python3 -c "
359
+ import json
360
+ with open('knowledge/n8n/n8n_embeddings.json') as f:
361
+ data = json.load(f)
362
+ print(f'Embeddings loaded: {len(data)} documents')
363
+ "
364
+ ```
365
+
366
+ ### Recovery Procedures
367
+
368
+ **Emergency Restore**:
369
+ 1. Stop all services: `docker-compose down`
370
+ 2. Restore from latest backup: `./scripts/restore.sh [backup-name]`
371
+ 3. Restart services: `docker-compose up -d`
372
+ 4. Verify functionality: Access web interface
373
+
374
+ **Database Recovery**:
375
+ ```sql
376
+ -- Check database integrity
377
+ SELECT schemaname, tablename, n_tup_ins, n_tup_upd, n_tup_del
378
+ FROM pg_stat_user_tables
379
+ WHERE schemaname = 'public';
380
+
381
+ -- Rebuild vector indexes if needed
382
+ REINDEX INDEX idx_embeddings_vector;
383
+ ```
384
+
385
+ ## 📈 Scaling
386
+
387
+ ### Horizontal Scaling
388
+
389
+ - Multiple n8n instances with queue mode
390
+ - Load balancer configuration
391
+ - Distributed vector store
392
+
393
+ ### Performance Monitoring
394
+
395
+ - Enable n8n metrics: `N8N_METRICS=true`
396
+ - Database query monitoring
397
+ - Vector search performance tracking
398
+ - Container resource utilization
399
+
400
+ ## 🔄 CI/CD Pipeline
401
+
402
+ ### Workflow Triggers
403
+
404
+ - **Push to main**: Automatic deployment
405
+ - **Scheduled**: Daily backups and knowledge sync
406
+ - **Manual**: On-demand operations via GitHub Actions
407
+
408
+ ### Pipeline Stages
409
+
410
+ 1. **Build**: Docker image creation and testing
411
+ 2. **Test**: Health checks and validation
412
+ 3. **Deploy**: Hugging Face Spaces deployment
413
+ 4. **Monitor**: Post-deployment verification
414
+
415
+ ## 📝 Contributing
416
+
417
+ 1. Fork the repository
418
+ 2. Create feature branch: `git checkout -b feature/new-capability`
419
+ 3. Commit changes: `git commit -am 'Add new capability'`
420
+ 4. Push branch: `git push origin feature/new-capability`
421
+ 5. Submit pull request
422
+
423
+ ### Development Workflow
424
+
425
+ ```bash
426
+ # Local development
427
+ docker-compose -f docker/docker-compose.yml up --build
428
+
429
+ # Run tests
430
+ ./scripts/test-infrastructure.sh
431
+
432
+ # Deploy to staging
433
+ gh workflow run deploy-to-hf.yml -f force_deploy=true
434
+ ```
435
+
436
+ ## 📞 Support
437
+
438
+ - **Issues**: GitHub Issues
439
+ - **Documentation**: [n8n Documentation](https://docs.n8n.io)
440
+ - **Community**: [n8n Community](https://community.n8n.io)
441
+
442
+ ## 📄 License
443
+
444
+ This project is licensed under the Apache License 2.0 - see the LICENSE file for details.
445
+
446
+ ---
447
+
448
+ **⚡ Pro Tips**:
449
+
450
+ 1. **Performance**: Use queue mode for high-volume workflows
451
+ 2. **Security**: Regular credential rotation and access reviews
452
+ 3. **Monitoring**: Set up alerts for failed workflows and system health
453
+ 4. **Backup**: Test restore procedures regularly
454
+ 5. **Knowledge**: Keep your knowledge base updated with latest best practices
455
+
456
+ ---
457
+
458
+ *Built with ❤️ for the n8n automation community*
SECURITY.md ADDED
@@ -0,0 +1,147 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Security Policy
2
+
3
+ ## Supported Versions
4
+
5
+ We actively maintain and provide security updates for the following versions:
6
+
7
+ | Version | Supported |
8
+ | ------- | ------------------ |
9
+ | 1.0.x | :white_check_mark: |
10
+
11
+ ## Reporting a Vulnerability
12
+
13
+ We take security vulnerabilities seriously. If you discover a security vulnerability in this n8n infrastructure, please report it responsibly.
14
+
15
+ ### How to Report
16
+
17
+ 1. **Do NOT** open a public GitHub issue for security vulnerabilities
18
+ 2. Send an email to: [email protected] (replace with your actual security contact)
19
+ 3. Include the following information:
20
+ - Description of the vulnerability
21
+ - Steps to reproduce the issue
22
+ - Potential impact assessment
23
+ - Suggested fix (if available)
24
+
25
+ ### What to Expect
26
+
27
+ - **Acknowledgment**: Within 48 hours of your report
28
+ - **Initial Assessment**: Within 7 days
29
+ - **Fix Timeline**: Critical issues within 14 days, others within 30 days
30
+ - **Disclosure**: Coordinated disclosure after fix is released
31
+
32
+ ## Security Best Practices
33
+
34
+ ### For Administrators
35
+
36
+ 1. **Environment Variables**:
37
+ - Never commit `.env` files to version control
38
+ - Use GitHub Secrets for all sensitive data
39
+ - Rotate encryption keys regularly (quarterly recommended)
40
+
41
+ 2. **Database Security**:
42
+ - Always use SSL connections to Supabase
43
+ - Enable Row Level Security (RLS) policies
44
+ - Regular backup encryption validation
45
+ - Monitor for unusual database activity
46
+
47
+ 3. **Container Security**:
48
+ - Keep n8n version pinned and updated
49
+ - Regular security scanning of Docker images
50
+ - Use non-root user inside containers
51
+ - Limit container network access
52
+
53
+ 4. **Access Control**:
54
+ - Enable n8n user management
55
+ - Use strong JWT secrets
56
+ - Implement webhook authentication
57
+ - Regular access review and cleanup
58
+
59
+ ### For Developers
60
+
61
+ 1. **Code Security**:
62
+ - No hardcoded credentials in source code
63
+ - Validate all webhook inputs
64
+ - Sanitize user inputs in workflows
65
+ - Use prepared statements for database queries
66
+
67
+ 2. **Workflow Security**:
68
+ - Audit workflow permissions regularly
69
+ - Secure credential storage in n8n
70
+ - Validate external API responses
71
+ - Implement proper error handling
72
+
73
+ 3. **AI Integration Security**:
74
+ - Validate AI model outputs
75
+ - Sanitize prompts and inputs
76
+ - Secure API key management
77
+ - Monitor AI usage and costs
78
+
79
+ ## Security Checklist
80
+
81
+ ### Pre-Deployment
82
+ - [ ] All secrets configured in GitHub repository
83
+ - [ ] Database SSL enforcement enabled
84
+ - [ ] Container security scan passed
85
+ - [ ] Webhook authentication configured
86
+ - [ ] Network security policies reviewed
87
+
88
+ ### Post-Deployment
89
+ - [ ] Health monitoring enabled
90
+ - [ ] Backup encryption verified
91
+ - [ ] Access logs configured
92
+ - [ ] Incident response plan ready
93
+ - [ ] Security contact information updated
94
+
95
+ ### Regular Maintenance
96
+ - [ ] Monthly security updates applied
97
+ - [ ] Quarterly credential rotation
98
+ - [ ] Backup integrity verification
99
+ - [ ] Security audit review
100
+ - [ ] Vulnerability scanning
101
+
102
+ ## Known Security Considerations
103
+
104
+ 1. **Hugging Face Spaces**: Public spaces expose the application URL. Use authentication and access controls.
105
+
106
+ 2. **Vector Embeddings**: Knowledge base content may contain sensitive information. Review before indexing.
107
+
108
+ 3. **Webhook Endpoints**: Publicly accessible URLs should implement proper authentication.
109
+
110
+ 4. **Database Access**: Ensure Supabase RLS policies are properly configured for your use case.
111
+
112
+ ## Incident Response
113
+
114
+ In case of a security incident:
115
+
116
+ 1. **Immediate Actions**:
117
+ - Disable affected services if necessary
118
+ - Preserve logs and evidence
119
+ - Assess scope and impact
120
+
121
+ 2. **Communication**:
122
+ - Notify security team immediately
123
+ - Prepare user communication if needed
124
+ - Coordinate with stakeholders
125
+
126
+ 3. **Recovery**:
127
+ - Apply security patches
128
+ - Restore from clean backups if needed
129
+ - Verify system integrity
130
+ - Update security measures
131
+
132
+ ## Security Resources
133
+
134
+ - [n8n Security Documentation](https://docs.n8n.io/security/)
135
+ - [Supabase Security Guide](https://supabase.com/docs/guides/platform/security)
136
+ - [Docker Security Best Practices](https://docs.docker.com/develop/security-best-practices/)
137
+ - [GitHub Actions Security](https://docs.github.com/en/actions/security-guides)
138
+
139
+ ## Contact
140
+
141
+ For security-related questions or concerns:
142
+ - Email: [email protected]
143
+ - Security Team: @security-team (GitHub)
144
+
145
+ ---
146
+
147
+ *Last updated: January 2025*
config/.env.example ADDED
@@ -0,0 +1,58 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # n8n Infrastructure Environment Configuration
2
+ # Copy this file to .env and fill in your actual values
3
+
4
+ # ===== CORE N8N CONFIGURATION =====
5
+ N8N_ENCRYPTION_KEY=your-32-character-encryption-key-here
6
+ N8N_USER_MANAGEMENT_JWT_SECRET=your-jwt-secret-key-here
7
+ N8N_HOST=your-n8n-domain.com
8
+ WEBHOOK_URL=https://your-n8n-domain.com
9
+
10
+ # ===== DATABASE CONFIGURATION =====
11
+ DB_TYPE=postgresdb
12
+ DB_POSTGRESDB_HOST=your-supabase-host.supabase.co
13
+ DB_POSTGRESDB_PORT=5432
14
+ DB_POSTGRESDB_DATABASE=postgres
15
+ DB_POSTGRESDB_USER=postgres
16
+ DB_POSTGRESDB_PASSWORD=your-supabase-password
17
+ DB_POSTGRESDB_SSL=true
18
+
19
+ # ===== DEPLOYMENT CONFIGURATION =====
20
+ HF_TOKEN=your-hugging-face-token
21
+ HF_SPACE_NAME=your-username/n8n-automation
22
+ GITHUB_TOKEN=your-github-token
23
+
24
+ # ===== AI INTEGRATIONS =====
25
+ GOOGLE_PROJECT_ID=your-google-cloud-project-id
26
+ GOOGLE_CREDENTIALS_PATH=/home/node/.n8n/credentials/google-service-account.json
27
+ OPENAI_API_KEY=sk-your-openai-api-key
28
+ ANTHROPIC_API_KEY=sk-ant-your-anthropic-key
29
+ VERTEX_AI_PROJECT=your-vertex-ai-project
30
+ VERTEX_AI_LOCATION=us-central1
31
+
32
+ # ===== VECTOR STORE CONFIGURATION =====
33
+ CHROMA_AUTH_TOKEN=your-chroma-auth-token
34
+ CHROMA_HOST=vector-store
35
+ CHROMA_PORT=8000
36
+
37
+ # ===== KNOWLEDGE BASE SYNC =====
38
+ KB_REPO_N8N=https://github.com/danilonovaisv/CHATGPT-knowledge-base.git
39
+ KB_BRANCH_N8N=main
40
+ KB_PATH_N8N=projects/n8n
41
+ KB_PATH_VIDEOS=projects/videos-e-animacoes
42
+ KB_PATH_MIDJOURNEY=projects/midjorney-prompt
43
+
44
+ # ===== MONITORING AND LOGGING =====
45
+ N8N_LOG_LEVEL=info
46
+ N8N_METRICS=true
47
+ SENTRY_DSN=your-sentry-dsn (optional)
48
+
49
+ # ===== BACKUP CONFIGURATION =====
50
+ BACKUP_SCHEDULE=0 2 * * *
51
+ BACKUP_RETENTION_DAYS=30
52
+ BACKUP_ENCRYPTION_PASSWORD=your-backup-encryption-password
53
+
54
+ # ===== SECURITY =====
55
+ ALLOWED_ORIGINS=https://your-domain.com,https://your-n8n-domain.com
56
+ CSRF_SECRET=your-csrf-secret
57
+ RATE_LIMIT_WINDOW=15
58
+ RATE_LIMIT_MAX=100
config/custom-nodes.json ADDED
@@ -0,0 +1,44 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "name": "n8n-custom-configuration",
3
+ "version": "1.0.0",
4
+ "description": "Custom configuration for n8n automation platform",
5
+ "customNodes": [
6
+ {
7
+ "name": "@n8n/n8n-nodes-langchain",
8
+ "version": "latest",
9
+ "description": "LangChain integration nodes for AI workflows"
10
+ },
11
+ {
12
+ "name": "n8n-nodes-google-vertex-ai",
13
+ "version": "latest",
14
+ "description": "Google Vertex AI integration"
15
+ },
16
+ {
17
+ "name": "n8n-nodes-supabase",
18
+ "version": "latest",
19
+ "description": "Supabase database operations"
20
+ },
21
+ {
22
+ "name": "n8n-nodes-vector-store",
23
+ "version": "latest",
24
+ "description": "Vector database operations for embeddings"
25
+ }
26
+ ],
27
+ "credentialTypes": [
28
+ {
29
+ "name": "googleCloudApi",
30
+ "displayName": "Google Cloud API",
31
+ "documentationUrl": "https://cloud.google.com/docs/authentication"
32
+ },
33
+ {
34
+ "name": "supabaseApi",
35
+ "displayName": "Supabase API",
36
+ "documentationUrl": "https://supabase.com/docs/guides/api"
37
+ },
38
+ {
39
+ "name": "openAiApi",
40
+ "displayName": "OpenAI API",
41
+ "documentationUrl": "https://platform.openai.com/docs/api-reference"
42
+ }
43
+ ]
44
+ }
docker/Dockerfile ADDED
@@ -0,0 +1,56 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Production-ready n8n Dockerfile for Hugging Face Spaces
2
+ FROM n8nio/n8n:1.17.1
3
+
4
+ # Set environment for production
5
+ ENV NODE_ENV=production
6
+ ENV N8N_PORT=7860
7
+ ENV N8N_LISTEN_ADDRESS=0.0.0.0
8
+ ENV N8N_PROTOCOL=https
9
+ ENV N8N_METRICS=true
10
+ ENV N8N_LOG_LEVEL=info
11
+ ENV N8N_LOG_OUTPUT=console
12
+ ENV EXECUTIONS_MODE=queue
13
+ ENV N8N_DISABLE_UI=false
14
+
15
+ # Create app user for security
16
+ USER root
17
+
18
+ # Install additional dependencies for AI and automation
19
+ RUN apk add --no-cache \
20
+ curl \
21
+ git \
22
+ python3 \
23
+ py3-pip \
24
+ postgresql-client \
25
+ && pip3 install --no-cache-dir \
26
+ langchain \
27
+ chromadb \
28
+ sentence-transformers
29
+
30
+ # Create necessary directories
31
+ RUN mkdir -p /home/node/.n8n/custom \
32
+ && mkdir -p /home/node/.n8n/nodes \
33
+ && mkdir -p /home/node/.n8n/credentials \
34
+ && mkdir -p /home/node/knowledge
35
+
36
+ # Copy custom configurations
37
+ COPY config/custom-nodes.json /home/node/.n8n/custom/
38
+ COPY knowledge/ /home/node/knowledge/
39
+
40
+ # Set proper permissions
41
+ RUN chown -R node:node /home/node/.n8n \
42
+ && chown -R node:node /home/node/knowledge \
43
+ && chmod +x /home/node/knowledge/sync-*.sh
44
+
45
+ # Switch back to node user
46
+ USER node
47
+
48
+ # Health check
49
+ HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 \
50
+ CMD curl -f http://localhost:7860/healthz || exit 1
51
+
52
+ # Expose port for Hugging Face Spaces
53
+ EXPOSE 7860
54
+
55
+ # Start n8n
56
+ CMD ["n8n", "start"]
docker/docker-compose.yml ADDED
@@ -0,0 +1,101 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ version: '3.8'
2
+
3
+ services:
4
+ n8n:
5
+ build:
6
+ context: ..
7
+ dockerfile: docker/Dockerfile
8
+ container_name: n8n-automation
9
+ restart: unless-stopped
10
+ ports:
11
+ - "7860:7860"
12
+ environment:
13
+ # Core Configuration
14
+ - NODE_ENV=production
15
+ - N8N_PORT=7860
16
+ - N8N_LISTEN_ADDRESS=0.0.0.0
17
+ - N8N_PROTOCOL=https
18
+ - N8N_HOST=${N8N_HOST:-localhost}
19
+
20
+ # Security
21
+ - N8N_ENCRYPTION_KEY=${N8N_ENCRYPTION_KEY}
22
+ - N8N_USER_MANAGEMENT_JWT_SECRET=${N8N_USER_MANAGEMENT_JWT_SECRET}
23
+
24
+ # Database Configuration
25
+ - DB_TYPE=postgresdb
26
+ - DB_POSTGRESDB_HOST=${DB_POSTGRESDB_HOST}
27
+ - DB_POSTGRESDB_PORT=${DB_POSTGRESDB_PORT:-5432}
28
+ - DB_POSTGRESDB_DATABASE=${DB_POSTGRESDB_DATABASE}
29
+ - DB_POSTGRESDB_USER=${DB_POSTGRESDB_USER}
30
+ - DB_POSTGRESDB_PASSWORD=${DB_POSTGRESDB_PASSWORD}
31
+ - DB_POSTGRESDB_SSL=true
32
+
33
+ # Webhooks and External Access
34
+ - WEBHOOK_URL=${WEBHOOK_URL}
35
+ - N8N_EDITOR_BASE_URL=${WEBHOOK_URL}
36
+
37
+ # AI and External Integrations
38
+ - GOOGLE_PROJECT_ID=${GOOGLE_PROJECT_ID}
39
+ - GOOGLE_CREDENTIALS_PATH=${GOOGLE_CREDENTIALS_PATH}
40
+ - OPENAI_API_KEY=${OPENAI_API_KEY}
41
+ - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
42
+
43
+ # Performance
44
+ - EXECUTIONS_MODE=queue
45
+ - EXECUTIONS_DATA_SAVE_ON_ERROR=all
46
+ - EXECUTIONS_DATA_SAVE_ON_SUCCESS=all
47
+ - EXECUTIONS_DATA_SAVE_MANUAL_EXECUTIONS=true
48
+
49
+ # Logging and Monitoring
50
+ - N8N_LOG_LEVEL=info
51
+ - N8N_LOG_OUTPUT=console
52
+ - N8N_METRICS=true
53
+
54
+ volumes:
55
+ - n8n_data:/home/node/.n8n
56
+ - ./knowledge:/home/node/knowledge:ro
57
+ - ./config/credentials:/home/node/.n8n/credentials:ro
58
+ - ./workflows/backup:/home/node/.n8n/backup
59
+
60
+ networks:
61
+ - n8n-network
62
+
63
+ depends_on:
64
+ - vector-store
65
+ - backup-service
66
+
67
+ vector-store:
68
+ image: chromadb/chroma:latest
69
+ container_name: n8n-vector-store
70
+ restart: unless-stopped
71
+ ports:
72
+ - "8000:8000"
73
+ environment:
74
+ - CHROMA_SERVER_AUTHN_CREDENTIALS=${CHROMA_AUTH_TOKEN}
75
+ - CHROMA_SERVER_AUTHN_PROVIDER=chromadb.auth.token.TokenAuthServerProvider
76
+ volumes:
77
+ - vector_data:/chroma/chroma
78
+ networks:
79
+ - n8n-network
80
+
81
+ backup-service:
82
+ image: alpine:latest
83
+ container_name: n8n-backup
84
+ restart: "no"
85
+ command: /bin/sh -c "while true; do sleep 3600; done"
86
+ volumes:
87
+ - n8n_data:/backup/n8n:ro
88
+ - ./scripts:/scripts:ro
89
+ - ./workflows/backup:/backup/workflows
90
+ networks:
91
+ - n8n-network
92
+
93
+ volumes:
94
+ n8n_data:
95
+ driver: local
96
+ vector_data:
97
+ driver: local
98
+
99
+ networks:
100
+ n8n-network:
101
+ driver: bridge
knowledge/.DS_Store ADDED
Binary file (6.15 kB). View file
 
package.json ADDED
@@ -0,0 +1,48 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "name": "n8n-infrastructure",
3
+ "version": "1.0.0",
4
+ "description": "Production-ready n8n infrastructure with AI integrations and automated knowledge management",
5
+ "private": true,
6
+ "keywords": [
7
+ "n8n",
8
+ "workflow-automation",
9
+ "ai-integration",
10
+ "langchain",
11
+ "vector-database",
12
+ "hugging-face",
13
+ "supabase",
14
+ "devops"
15
+ ],
16
+ "scripts": {
17
+ "dev": "docker compose -f docker/docker-compose.yml up --build",
18
+ "start": "docker compose -f docker/docker-compose.yml up -d",
19
+ "stop": "docker compose -f docker/docker-compose.yml down",
20
+ "logs": "docker compose -f docker/docker-compose.yml logs -f n8n",
21
+ "backup": "bash scripts/backup.sh",
22
+ "restore": "bash scripts/restore.sh",
23
+ "sync-knowledge": "bash scripts/sync-knowledge.sh",
24
+ "health-check": "curl -f http://localhost:7860/healthz",
25
+ "clean": "docker compose -f docker/docker-compose.yml down -v && docker system prune -f",
26
+ "deploy": "gh workflow run deploy-to-hf.yml",
27
+ "test": "bash scripts/test-infrastructure.sh"
28
+ },
29
+ "repository": {
30
+ "type": "git",
31
+ "url": "https://github.com/your-username/n8n-infra.git"
32
+ },
33
+ "author": {
34
+ "name": "Your Name",
35
+ "email": "[email protected]"
36
+ },
37
+ "license": "Apache-2.0",
38
+ "engines": {
39
+ "node": ">=18.0.0",
40
+ "npm": ">=9.0.0"
41
+ },
42
+ "devDependencies": {
43
+ "@types/node": "^20.0.0"
44
+ },
45
+ "dependencies": {
46
+ "huggingface_hub": "^0.20.1"
47
+ }
48
+ }
scripts/backup.sh ADDED
@@ -0,0 +1,198 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/bin/bash
2
+
3
+ # n8n Infrastructure Backup Script
4
+ # Backs up workflows, credentials, and configurations
5
+ # Usage: ./backup.sh [backup-name]
6
+
7
+ set -euo pipefail
8
+
9
+ # Configuration
10
+ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
11
+ PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
12
+ BACKUP_DIR="$PROJECT_ROOT/workflows/backup"
13
+ TIMESTAMP=$(date +"%Y%m%d_%H%M%S")
14
+ BACKUP_NAME="${1:-n8n_backup_$TIMESTAMP}"
15
+
16
+ # Colors for output
17
+ RED='\033[0;31m'
18
+ GREEN='\033[0;32m'
19
+ YELLOW='\033[1;33m'
20
+ NC='\033[0m' # No Color
21
+
22
+ log_info() {
23
+ echo -e "${GREEN}[INFO]${NC} $1"
24
+ }
25
+
26
+ log_warn() {
27
+ echo -e "${YELLOW}[WARN]${NC} $1"
28
+ }
29
+
30
+ log_error() {
31
+ echo -e "${RED}[ERROR]${NC} $1"
32
+ }
33
+
34
+ # Check if Docker is running
35
+ check_docker() {
36
+ if ! docker ps > /dev/null 2>&1; then
37
+ log_error "Docker is not running or accessible"
38
+ exit 1
39
+ fi
40
+ }
41
+
42
+ # Create backup directory
43
+ create_backup_dir() {
44
+ local backup_path="$BACKUP_DIR/$BACKUP_NAME"
45
+ mkdir -p "$backup_path"
46
+ echo "$backup_path"
47
+ }
48
+
49
+ # Backup n8n workflows via API
50
+ backup_workflows() {
51
+ local backup_path="$1"
52
+ local container_name="n8n-automation"
53
+
54
+ log_info "Backing up n8n workflows..."
55
+
56
+ if docker ps --format '{{.Names}}' | grep -q "^$container_name$"; then
57
+ # Export workflows using n8n CLI inside container
58
+ docker exec "$container_name" n8n export:workflow --all --output="/home/node/.n8n/backup/workflows_$TIMESTAMP.json" || {
59
+ log_warn "Failed to export workflows via CLI, trying API approach"
60
+ return 1
61
+ }
62
+
63
+ # Copy exported file to backup directory
64
+ docker cp "$container_name:/home/node/.n8n/backup/workflows_$TIMESTAMP.json" "$backup_path/"
65
+ log_info "Workflows backed up successfully"
66
+ else
67
+ log_error "n8n container not found or not running"
68
+ return 1
69
+ fi
70
+ }
71
+
72
+ # Backup credentials (encrypted)
73
+ backup_credentials() {
74
+ local backup_path="$1"
75
+ local container_name="n8n-automation"
76
+
77
+ log_info "Backing up encrypted credentials..."
78
+
79
+ if docker ps --format '{{.Names}}' | grep -q "^$container_name$"; then
80
+ docker cp "$container_name:/home/node/.n8n/credentials" "$backup_path/" 2>/dev/null || {
81
+ log_warn "No credentials found or access denied"
82
+ }
83
+ fi
84
+ }
85
+
86
+ # Backup database schema and essential data
87
+ backup_database() {
88
+ local backup_path="$1"
89
+
90
+ log_info "Backing up database schema..."
91
+
92
+ if [[ -n "${DB_POSTGRESDB_PASSWORD:-}" ]]; then
93
+ export PGPASSWORD="$DB_POSTGRESDB_PASSWORD"
94
+
95
+ pg_dump \
96
+ --host="${DB_POSTGRESDB_HOST}" \
97
+ --port="${DB_POSTGRESDB_PORT:-5432}" \
98
+ --username="${DB_POSTGRESDB_USER}" \
99
+ --dbname="${DB_POSTGRESDB_DATABASE}" \
100
+ --schema-only \
101
+ --no-owner \
102
+ --no-privileges \
103
+ > "$backup_path/schema_$TIMESTAMP.sql" || {
104
+ log_error "Database backup failed"
105
+ return 1
106
+ }
107
+
108
+ log_info "Database schema backed up successfully"
109
+ else
110
+ log_warn "Database credentials not available, skipping database backup"
111
+ fi
112
+ }
113
+
114
+ # Backup knowledge base content
115
+ backup_knowledge() {
116
+ local backup_path="$1"
117
+
118
+ log_info "Backing up knowledge base..."
119
+
120
+ if [[ -d "$PROJECT_ROOT/knowledge" ]]; then
121
+ cp -r "$PROJECT_ROOT/knowledge" "$backup_path/"
122
+ log_info "Knowledge base backed up successfully"
123
+ else
124
+ log_warn "Knowledge base directory not found"
125
+ fi
126
+ }
127
+
128
+ # Create backup metadata
129
+ create_metadata() {
130
+ local backup_path="$1"
131
+
132
+ cat > "$backup_path/backup_metadata.json" << EOF
133
+ {
134
+ "backup_name": "$BACKUP_NAME",
135
+ "timestamp": "$TIMESTAMP",
136
+ "created_at": "$(date -Iseconds)",
137
+ "n8n_version": "$(docker exec n8n-automation n8n --version 2>/dev/null || echo 'unknown')",
138
+ "backup_type": "full",
139
+ "components": [
140
+ "workflows",
141
+ "credentials",
142
+ "database_schema",
143
+ "knowledge_base"
144
+ ],
145
+ "notes": "Automated backup created by backup.sh script"
146
+ }
147
+ EOF
148
+ }
149
+
150
+ # Cleanup old backups
151
+ cleanup_old_backups() {
152
+ local retention_days="${BACKUP_RETENTION_DAYS:-30}"
153
+
154
+ log_info "Cleaning up backups older than $retention_days days..."
155
+
156
+ find "$BACKUP_DIR" -type d -name "n8n_backup_*" -mtime "+$retention_days" -exec rm -rf {} + 2>/dev/null || true
157
+ }
158
+
159
+ # Main backup process
160
+ main() {
161
+ log_info "Starting n8n infrastructure backup: $BACKUP_NAME"
162
+
163
+ # Preliminary checks
164
+ check_docker
165
+
166
+ # Load environment variables
167
+ if [[ -f "$PROJECT_ROOT/.env" ]]; then
168
+ source "$PROJECT_ROOT/.env"
169
+ fi
170
+
171
+ # Create backup directory
172
+ local backup_path=$(create_backup_dir)
173
+ log_info "Backup directory created: $backup_path"
174
+
175
+ # Perform backups
176
+ backup_workflows "$backup_path" || log_warn "Workflow backup incomplete"
177
+ backup_credentials "$backup_path" || log_warn "Credentials backup incomplete"
178
+ backup_database "$backup_path" || log_warn "Database backup incomplete"
179
+ backup_knowledge "$backup_path" || log_warn "Knowledge base backup incomplete"
180
+
181
+ # Create metadata
182
+ create_metadata "$backup_path"
183
+
184
+ # Cleanup old backups
185
+ cleanup_old_backups
186
+
187
+ log_info "Backup completed successfully: $backup_path"
188
+
189
+ # Optional: Create compressed archive
190
+ if command -v tar > /dev/null; then
191
+ local archive_name="$BACKUP_DIR/${BACKUP_NAME}.tar.gz"
192
+ tar -czf "$archive_name" -C "$BACKUP_DIR" "$BACKUP_NAME"
193
+ log_info "Compressed backup created: $archive_name"
194
+ fi
195
+ }
196
+
197
+ # Run main function
198
+ main "$@"
scripts/restore.sh ADDED
@@ -0,0 +1,204 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/bin/bash
2
+
3
+ # n8n Infrastructure Restore Script
4
+ # Restores workflows, credentials, and configurations from backup
5
+ # Usage: ./restore.sh <backup-name>
6
+
7
+ set -euo pipefail
8
+
9
+ # Configuration
10
+ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
11
+ PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
12
+ BACKUP_DIR="$PROJECT_ROOT/workflows/backup"
13
+
14
+ # Colors for output
15
+ RED='\033[0;31m'
16
+ GREEN='\033[0;32m'
17
+ YELLOW='\033[1;33m'
18
+ NC='\033[0m' # No Color
19
+
20
+ log_info() {
21
+ echo -e "${GREEN}[INFO]${NC} $1"
22
+ }
23
+
24
+ log_warn() {
25
+ echo -e "${YELLOW}[WARN]${NC} $1"
26
+ }
27
+
28
+ log_error() {
29
+ echo -e "${RED}[ERROR]${NC} $1"
30
+ }
31
+
32
+ # Validate backup name argument
33
+ if [[ $# -eq 0 ]]; then
34
+ log_error "Usage: $0 <backup-name>"
35
+ log_info "Available backups:"
36
+ ls -1 "$BACKUP_DIR" | grep -E "^n8n_backup_" | head -10
37
+ exit 1
38
+ fi
39
+
40
+ BACKUP_NAME="$1"
41
+ BACKUP_PATH="$BACKUP_DIR/$BACKUP_NAME"
42
+
43
+ # Check if backup exists
44
+ if [[ ! -d "$BACKUP_PATH" ]]; then
45
+ # Try with .tar.gz extension
46
+ if [[ -f "$BACKUP_DIR/$BACKUP_NAME.tar.gz" ]]; then
47
+ log_info "Found compressed backup, extracting..."
48
+ tar -xzf "$BACKUP_DIR/$BACKUP_NAME.tar.gz" -C "$BACKUP_DIR"
49
+ else
50
+ log_error "Backup not found: $BACKUP_NAME"
51
+ exit 1
52
+ fi
53
+ fi
54
+
55
+ # Verify backup integrity
56
+ verify_backup() {
57
+ local metadata_file="$BACKUP_PATH/backup_metadata.json"
58
+
59
+ if [[ ! -f "$metadata_file" ]]; then
60
+ log_warn "Backup metadata not found, proceeding with caution"
61
+ return 0
62
+ fi
63
+
64
+ log_info "Backup verification:"
65
+ cat "$metadata_file" | jq -r '.backup_name, .created_at, .backup_type'
66
+ }
67
+
68
+ # Restore workflows
69
+ restore_workflows() {
70
+ local container_name="n8n-automation"
71
+
72
+ log_info "Restoring workflows..."
73
+
74
+ if ! docker ps --format '{{.Names}}' | grep -q "^$container_name$"; then
75
+ log_error "n8n container not running. Start the container first."
76
+ return 1
77
+ fi
78
+
79
+ # Find workflow backup file
80
+ local workflow_file=$(find "$BACKUP_PATH" -name "workflows_*.json" | head -1)
81
+
82
+ if [[ -n "$workflow_file" ]]; then
83
+ # Copy workflow file to container
84
+ docker cp "$workflow_file" "$container_name:/tmp/workflows_restore.json"
85
+
86
+ # Import workflows
87
+ docker exec "$container_name" n8n import:workflow --input="/tmp/workflows_restore.json" || {
88
+ log_error "Failed to import workflows"
89
+ return 1
90
+ }
91
+
92
+ log_info "Workflows restored successfully"
93
+ else
94
+ log_warn "No workflow backup file found"
95
+ fi
96
+ }
97
+
98
+ # Restore credentials
99
+ restore_credentials() {
100
+ local container_name="n8n-automation"
101
+
102
+ log_info "Restoring credentials..."
103
+
104
+ if [[ -d "$BACKUP_PATH/credentials" ]]; then
105
+ docker cp "$BACKUP_PATH/credentials/." "$container_name:/home/node/.n8n/credentials/"
106
+ log_info "Credentials restored successfully"
107
+ else
108
+ log_warn "No credentials backup found"
109
+ fi
110
+ }
111
+
112
+ # Restore database (schema only, data should be preserved)
113
+ restore_database() {
114
+ log_info "Restoring database schema..."
115
+
116
+ local schema_file=$(find "$BACKUP_PATH" -name "schema_*.sql" | head -1)
117
+
118
+ if [[ -n "$schema_file" && -n "${DB_POSTGRESDB_PASSWORD:-}" ]]; then
119
+ export PGPASSWORD="$DB_POSTGRESDB_PASSWORD"
120
+
121
+ log_warn "This will update database schema. Proceed? (y/N)"
122
+ read -r response
123
+ if [[ "$response" =~ ^[Yy]$ ]]; then
124
+ psql \
125
+ --host="${DB_POSTGRESDB_HOST}" \
126
+ --port="${DB_POSTGRESDB_PORT:-5432}" \
127
+ --username="${DB_POSTGRESDB_USER}" \
128
+ --dbname="${DB_POSTGRESDB_DATABASE}" \
129
+ --file="$schema_file" || {
130
+ log_error "Database restore failed"
131
+ return 1
132
+ }
133
+ log_info "Database schema restored successfully"
134
+ else
135
+ log_info "Database restore skipped"
136
+ fi
137
+ else
138
+ log_warn "No database backup found or credentials missing"
139
+ fi
140
+ }
141
+
142
+ # Restart services after restore
143
+ restart_services() {
144
+ log_info "Restarting n8n services..."
145
+
146
+ docker-compose -f "$PROJECT_ROOT/docker/docker-compose.yml" restart n8n
147
+
148
+ # Wait for service to be ready
149
+ local max_attempts=30
150
+ local attempt=1
151
+
152
+ while [[ $attempt -le $max_attempts ]]; do
153
+ if curl -f http://localhost:7860/healthz > /dev/null 2>&1; then
154
+ log_info "n8n service is ready"
155
+ break
156
+ fi
157
+
158
+ log_info "Waiting for n8n to start... ($attempt/$max_attempts)"
159
+ sleep 10
160
+ ((attempt++))
161
+ done
162
+
163
+ if [[ $attempt -gt $max_attempts ]]; then
164
+ log_error "n8n failed to start after restore"
165
+ return 1
166
+ fi
167
+ }
168
+
169
+ # Main restore process
170
+ main() {
171
+ log_info "Starting n8n infrastructure restore: $BACKUP_NAME"
172
+
173
+ # Load environment variables
174
+ if [[ -f "$PROJECT_ROOT/.env" ]]; then
175
+ source "$PROJECT_ROOT/.env"
176
+ fi
177
+
178
+ # Verify backup
179
+ verify_backup
180
+
181
+ # Confirm restore operation
182
+ log_warn "This will restore n8n configuration from backup: $BACKUP_NAME"
183
+ log_warn "Current workflows and credentials may be overwritten. Continue? (y/N)"
184
+ read -r response
185
+
186
+ if [[ ! "$response" =~ ^[Yy]$ ]]; then
187
+ log_info "Restore operation cancelled"
188
+ exit 0
189
+ fi
190
+
191
+ # Perform restore operations
192
+ restore_workflows || log_warn "Workflow restore incomplete"
193
+ restore_credentials || log_warn "Credentials restore incomplete"
194
+ restore_database || log_warn "Database restore incomplete"
195
+
196
+ # Restart services
197
+ restart_services
198
+
199
+ log_info "Restore completed successfully"
200
+ log_info "Access your n8n instance at: ${WEBHOOK_URL:-http://localhost:7860}"
201
+ }
202
+
203
+ # Run main function
204
+ main "$@"
scripts/sync-knowledge.sh ADDED
@@ -0,0 +1,367 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/bin/bash
2
+
3
+ # Knowledge Base Synchronization Script
4
+ # Syncs content from multiple GitHub repositories and generates embeddings
5
+ # Usage: ./sync-knowledge.sh
6
+
7
+ set -euo pipefail
8
+
9
+ # Configuration
10
+ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
11
+ PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
12
+ KNOWLEDGE_DIR="$PROJECT_ROOT/knowledge"
13
+ TEMP_DIR="/tmp/kb-sync-$$"
14
+
15
+ # Colors for output
16
+ RED='\033[0;31m'
17
+ GREEN='\033[0;32m'
18
+ YELLOW='\033[1;33m'
19
+ BLUE='\033[0;34m'
20
+ NC='\033[0m' # No Color
21
+
22
+ log_info() {
23
+ echo -e "${GREEN}[INFO]${NC} $1"
24
+ }
25
+
26
+ log_warn() {
27
+ echo -e "${YELLOW}[WARN]${NC} $1"
28
+ }
29
+
30
+ log_error() {
31
+ echo -e "${RED}[ERROR]${NC} $1"
32
+ }
33
+
34
+ log_debug() {
35
+ echo -e "${BLUE}[DEBUG]${NC} $1"
36
+ }
37
+
38
+ # Cleanup function
39
+ cleanup() {
40
+ log_info "Cleaning up temporary files..."
41
+ rm -rf "$TEMP_DIR"
42
+ }
43
+
44
+ # Set trap for cleanup
45
+ trap cleanup EXIT
46
+
47
+ # Check dependencies
48
+ check_dependencies() {
49
+ local deps=("git" "curl" "jq")
50
+
51
+ for dep in "${deps[@]}"; do
52
+ if ! command -v "$dep" > /dev/null; then
53
+ log_error "Required dependency not found: $dep"
54
+ exit 1
55
+ fi
56
+ done
57
+ }
58
+
59
+ # Load environment variables
60
+ load_env() {
61
+ if [[ -f "$PROJECT_ROOT/.env" ]]; then
62
+ source "$PROJECT_ROOT/.env"
63
+ else
64
+ log_error ".env file not found. Copy .env.example and configure it."
65
+ exit 1
66
+ fi
67
+ }
68
+
69
+ # Clone or update repository
70
+ sync_repository() {
71
+ local repo_url="$1"
72
+ local target_path="$2"
73
+ local branch="${3:-main}"
74
+ local subpath="$4"
75
+
76
+ log_info "Syncing repository: $repo_url"
77
+ log_debug "Target: $target_path, Branch: $branch, Subpath: $subpath"
78
+
79
+ local repo_name=$(basename "$repo_url" .git)
80
+ local temp_repo_path="$TEMP_DIR/$repo_name"
81
+
82
+ # Clone repository to temp directory
83
+ git clone --depth 1 --branch "$branch" "$repo_url" "$temp_repo_path" || {
84
+ log_error "Failed to clone repository: $repo_url"
85
+ return 1
86
+ }
87
+
88
+ # Copy specific subpath to target
89
+ local source_path="$temp_repo_path/$subpath"
90
+ if [[ -d "$source_path" ]]; then
91
+ mkdir -p "$(dirname "$target_path")"
92
+ cp -r "$source_path/." "$target_path/"
93
+ log_info "Successfully synced to: $target_path"
94
+ else
95
+ log_warn "Subpath not found: $subpath in $repo_url"
96
+ return 1
97
+ fi
98
+ }
99
+
100
+ # Generate embeddings for knowledge content
101
+ generate_embeddings() {
102
+ local knowledge_path="$1"
103
+ local collection_name="$2"
104
+
105
+ log_info "Generating embeddings for: $collection_name"
106
+
107
+ # Create Python script for embedding generation
108
+ cat > "$TEMP_DIR/generate_embeddings.py" << 'EOF'
109
+ import os
110
+ import json
111
+ import sys
112
+ from pathlib import Path
113
+ import hashlib
114
+ import requests
115
+ from sentence_transformers import SentenceTransformer
116
+
117
+ def load_model():
118
+ """Load sentence transformer model"""
119
+ try:
120
+ model = SentenceTransformer('all-MiniLM-L6-v2')
121
+ return model
122
+ except Exception as e:
123
+ print(f"Error loading model: {e}")
124
+ return None
125
+
126
+ def process_text_files(knowledge_path, collection_name):
127
+ """Process text files and generate embeddings"""
128
+ model = load_model()
129
+ if not model:
130
+ return False
131
+
132
+ embeddings_data = []
133
+ knowledge_path = Path(knowledge_path)
134
+
135
+ # Process markdown and text files
136
+ for file_path in knowledge_path.rglob("*.md"):
137
+ try:
138
+ with open(file_path, 'r', encoding='utf-8') as f:
139
+ content = f.read()
140
+
141
+ # Generate embedding
142
+ embedding = model.encode(content).tolist()
143
+
144
+ # Create document metadata
145
+ doc_id = hashlib.md5(str(file_path).encode()).hexdigest()
146
+
147
+ embeddings_data.append({
148
+ "id": doc_id,
149
+ "content": content,
150
+ "embedding": embedding,
151
+ "metadata": {
152
+ "file_path": str(file_path.relative_to(knowledge_path)),
153
+ "file_name": file_path.name,
154
+ "collection": collection_name,
155
+ "content_type": "markdown",
156
+ "size": len(content)
157
+ }
158
+ })
159
+
160
+ except Exception as e:
161
+ print(f"Error processing file {file_path}: {e}")
162
+
163
+ # Save embeddings to JSON file
164
+ output_file = knowledge_path / f"{collection_name}_embeddings.json"
165
+ with open(output_file, 'w', encoding='utf-8') as f:
166
+ json.dump(embeddings_data, f, indent=2, ensure_ascii=False)
167
+
168
+ print(f"Generated {len(embeddings_data)} embeddings for {collection_name}")
169
+ return True
170
+
171
+ if __name__ == "__main__":
172
+ if len(sys.argv) != 3:
173
+ print("Usage: python generate_embeddings.py <knowledge_path> <collection_name>")
174
+ sys.exit(1)
175
+
176
+ knowledge_path = sys.argv[1]
177
+ collection_name = sys.argv[2]
178
+
179
+ if process_text_files(knowledge_path, collection_name):
180
+ print("Embedding generation completed successfully")
181
+ else:
182
+ print("Embedding generation failed")
183
+ sys.exit(1)
184
+ EOF
185
+
186
+ # Run embedding generation
187
+ python3 "$TEMP_DIR/generate_embeddings.py" "$knowledge_path" "$collection_name" || {
188
+ log_error "Failed to generate embeddings for $collection_name"
189
+ return 1
190
+ }
191
+ }
192
+
193
+ # Upload embeddings to vector store
194
+ upload_embeddings() {
195
+ local embeddings_file="$1"
196
+ local collection_name="$2"
197
+
198
+ log_info "Uploading embeddings to vector store: $collection_name"
199
+
200
+ if [[ ! -f "$embeddings_file" ]]; then
201
+ log_error "Embeddings file not found: $embeddings_file"
202
+ return 1
203
+ fi
204
+
205
+ # Upload to ChromaDB
206
+ local chroma_url="http://${CHROMA_HOST:-localhost}:${CHROMA_PORT:-8000}"
207
+
208
+ curl -X POST "$chroma_url/api/v1/collections" \
209
+ -H "Content-Type: application/json" \
210
+ -H "Authorization: Bearer ${CHROMA_AUTH_TOKEN}" \
211
+ -d "{\"name\": \"$collection_name\"}" || true
212
+
213
+ # Process and upload embeddings in batches
214
+ python3 - << EOF
215
+ import json
216
+ import requests
217
+ import sys
218
+ from pathlib import Path
219
+
220
+ def upload_batch(embeddings_data, collection_name, chroma_url, auth_token):
221
+ """Upload embeddings in batches to ChromaDB"""
222
+ batch_size = 100
223
+ total_docs = len(embeddings_data)
224
+
225
+ for i in range(0, total_docs, batch_size):
226
+ batch = embeddings_data[i:i+batch_size]
227
+
228
+ # Prepare batch data for ChromaDB
229
+ ids = [doc["id"] for doc in batch]
230
+ embeddings = [doc["embedding"] for doc in batch]
231
+ metadatas = [doc["metadata"] for doc in batch]
232
+ documents = [doc["content"] for doc in batch]
233
+
234
+ payload = {
235
+ "ids": ids,
236
+ "embeddings": embeddings,
237
+ "metadatas": metadatas,
238
+ "documents": documents
239
+ }
240
+
241
+ try:
242
+ response = requests.post(
243
+ f"{chroma_url}/api/v1/collections/{collection_name}/add",
244
+ json=payload,
245
+ headers={
246
+ "Content-Type": "application/json",
247
+ "Authorization": f"Bearer {auth_token}"
248
+ },
249
+ timeout=30
250
+ )
251
+
252
+ if response.status_code == 200:
253
+ print(f"Uploaded batch {i//batch_size + 1} ({len(batch)} documents)")
254
+ else:
255
+ print(f"Error uploading batch {i//batch_size + 1}: {response.status_code}")
256
+ print(f"Response: {response.text}")
257
+
258
+ except Exception as e:
259
+ print(f"Error uploading batch {i//batch_size + 1}: {e}")
260
+ continue
261
+
262
+ # Load and upload embeddings
263
+ embeddings_file = "$embeddings_file"
264
+ collection_name = "$collection_name"
265
+ chroma_url = "$chroma_url"
266
+ auth_token = "${CHROMA_AUTH_TOKEN:-}"
267
+
268
+ try:
269
+ with open(embeddings_file, 'r', encoding='utf-8') as f:
270
+ embeddings_data = json.load(f)
271
+
272
+ upload_batch(embeddings_data, collection_name, chroma_url, auth_token)
273
+ print(f"Successfully uploaded {len(embeddings_data)} embeddings to {collection_name}")
274
+
275
+ except Exception as e:
276
+ print(f"Error: {e}")
277
+ sys.exit(1)
278
+ EOF
279
+ }
280
+
281
+ # Sync all knowledge repositories
282
+ sync_all_repositories() {
283
+ log_info "Starting knowledge base synchronization..."
284
+
285
+ mkdir -p "$TEMP_DIR"
286
+
287
+ # Repository configurations
288
+ declare -A repos=(
289
+ ["n8n"]="${KB_REPO_N8N:-}:${KB_PATH_N8N:-projects/n8n}"
290
+ ["videos-e-animacoes"]="${KB_REPO_N8N:-}:${KB_PATH_VIDEOS:-projects/videos-e-animacoes}"
291
+ ["midjourney-prompt"]="${KB_REPO_N8N:-}:${KB_PATH_MIDJOURNEY:-projects/midjorney-prompt}"
292
+ )
293
+
294
+ for collection in "${!repos[@]}"; do
295
+ local repo_config="${repos[$collection]}"
296
+ local repo_url=$(echo "$repo_config" | cut -d':' -f1)
297
+ local subpath=$(echo "$repo_config" | cut -d':' -f2)
298
+ local target_path="$KNOWLEDGE_DIR/$collection"
299
+
300
+ if [[ -n "$repo_url" ]]; then
301
+ log_info "Syncing collection: $collection"
302
+
303
+ # Sync repository
304
+ sync_repository "$repo_url" "$target_path" "${KB_BRANCH_N8N:-main}" "$subpath"
305
+
306
+ # Generate embeddings
307
+ generate_embeddings "$target_path" "$collection"
308
+
309
+ # Upload to vector store
310
+ local embeddings_file="$target_path/${collection}_embeddings.json"
311
+ if [[ -f "$embeddings_file" ]]; then
312
+ upload_embeddings "$embeddings_file" "$collection"
313
+ fi
314
+
315
+ else
316
+ log_warn "Repository URL not configured for collection: $collection"
317
+ fi
318
+ done
319
+ }
320
+
321
+ # Update n8n with new knowledge
322
+ update_n8n_knowledge() {
323
+ log_info "Notifying n8n of knowledge base updates..."
324
+
325
+ # Create a webhook trigger to refresh knowledge in n8n workflows
326
+ if [[ -n "${WEBHOOK_URL:-}" ]]; then
327
+ local webhook_endpoint="$WEBHOOK_URL/webhook/knowledge-sync"
328
+
329
+ curl -X POST "$webhook_endpoint" \
330
+ -H "Content-Type: application/json" \
331
+ -d "{\"event\": \"knowledge_updated\", \"timestamp\": \"$(date -Iseconds)\"}" \
332
+ > /dev/null 2>&1 || {
333
+ log_warn "Failed to notify n8n of knowledge updates"
334
+ }
335
+ fi
336
+ }
337
+
338
+ # Main synchronization process
339
+ main() {
340
+ log_info "Starting knowledge base synchronization"
341
+
342
+ # Preliminary checks
343
+ check_dependencies
344
+ load_env
345
+
346
+ # Create knowledge directories
347
+ mkdir -p "$KNOWLEDGE_DIR"/{n8n,videos-e-animacoes,midjourney-prompt}
348
+
349
+ # Sync all repositories
350
+ sync_all_repositories
351
+
352
+ # Update n8n
353
+ update_n8n_knowledge
354
+
355
+ log_info "Knowledge base synchronization completed"
356
+
357
+ # Generate summary
358
+ log_info "Synchronization Summary:"
359
+ find "$KNOWLEDGE_DIR" -name "*_embeddings.json" -exec basename {} \; | while read file; do
360
+ local collection=$(echo "$file" | sed 's/_embeddings.json//')
361
+ local count=$(jq '. | length' "$KNOWLEDGE_DIR/$collection/$file" 2>/dev/null || echo "0")
362
+ log_info " - $collection: $count documents"
363
+ done
364
+ }
365
+
366
+ # Run main function
367
+ main "$@"
scripts/test-infrastructure.sh ADDED
@@ -0,0 +1,190 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/bin/bash
2
+
3
+ # Infrastructure Testing Script
4
+ # Tests all components of the n8n infrastructure
5
+ # Usage: ./test-infrastructure.sh
6
+
7
+ set -euo pipefail
8
+
9
+ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
10
+ PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
11
+
12
+ # Colors
13
+ RED='\033[0;31m'
14
+ GREEN='\033[0;32m'
15
+ YELLOW='\033[1;33m'
16
+ BLUE='\033[0;34m'
17
+ NC='\033[0m'
18
+
19
+ log_info() { echo -e "${GREEN}[INFO]${NC} $1"; }
20
+ log_warn() { echo -e "${YELLOW}[WARN]${NC} $1"; }
21
+ log_error() { echo -e "${RED}[ERROR]${NC} $1"; }
22
+ log_test() { echo -e "${BLUE}[TEST]${NC} $1"; }
23
+
24
+ # Test results tracking
25
+ TESTS_TOTAL=0
26
+ TESTS_PASSED=0
27
+ TESTS_FAILED=0
28
+
29
+ run_test() {
30
+ local test_name="$1"
31
+ local test_command="$2"
32
+
33
+ ((TESTS_TOTAL++))
34
+ log_test "Running: $test_name"
35
+
36
+ if eval "$test_command" > /dev/null 2>&1; then
37
+ echo " ✅ PASSED"
38
+ ((TESTS_PASSED++))
39
+ return 0
40
+ else
41
+ echo " ❌ FAILED"
42
+ ((TESTS_FAILED++))
43
+ return 1
44
+ fi
45
+ }
46
+
47
+ # Load environment
48
+ if [[ -f "$PROJECT_ROOT/.env" ]]; then
49
+ source "$PROJECT_ROOT/.env"
50
+ fi
51
+
52
+ # Test Docker configuration
53
+ test_docker() {
54
+ log_info "Testing Docker configuration..."
55
+
56
+ run_test "Docker daemon accessible" "docker --version"
57
+ run_test "Docker Compose available" "docker-compose --version"
58
+ run_test "Dockerfile syntax" "docker build -f docker/Dockerfile --dry-run . 2>/dev/null || docker build -f docker/Dockerfile -t n8n-test . --dry-run"
59
+ run_test "Docker Compose syntax" "docker-compose -f docker/docker-compose.yml config"
60
+ }
61
+
62
+ # Test environment configuration
63
+ test_environment() {
64
+ log_info "Testing environment configuration..."
65
+
66
+ run_test "Environment example exists" "[[ -f config/.env.example ]]"
67
+ run_test "Required directories exist" "[[ -d workflows && -d knowledge && -d scripts ]]"
68
+
69
+ if [[ -f .env ]]; then
70
+ run_test "Environment file loaded" "[[ -n \${N8N_ENCRYPTION_KEY:-} ]]"
71
+ run_test "Database config present" "[[ -n \${DB_POSTGRESDB_HOST:-} ]]"
72
+ else
73
+ log_warn "No .env file found - using example for testing"
74
+ fi
75
+ }
76
+
77
+ # Test scripts
78
+ test_scripts() {
79
+ log_info "Testing infrastructure scripts..."
80
+
81
+ run_test "Backup script executable" "[[ -x scripts/backup.sh ]]"
82
+ run_test "Restore script executable" "[[ -x scripts/restore.sh ]]"
83
+ run_test "Sync script executable" "[[ -x scripts/sync-knowledge.sh ]]"
84
+ run_test "Backup script syntax" "bash -n scripts/backup.sh"
85
+ run_test "Restore script syntax" "bash -n scripts/restore.sh"
86
+ run_test "Sync script syntax" "bash -n scripts/sync-knowledge.sh"
87
+ }
88
+
89
+ # Test GitHub Actions
90
+ test_github_actions() {
91
+ log_info "Testing GitHub Actions workflows..."
92
+
93
+ run_test "Deploy workflow exists" "[[ -f .github/workflows/deploy-to-hf.yml ]]"
94
+ run_test "Backup workflow exists" "[[ -f .github/workflows/backup-workflows.yml ]]"
95
+ run_test "Sync workflow exists" "[[ -f .github/workflows/sync-knowledge.yml ]]"
96
+
97
+ # Validate workflow syntax (basic YAML check)
98
+ if command -v python3 > /dev/null; then
99
+ run_test "Deploy workflow syntax" "python3 -c \"import yaml; yaml.safe_load(open('.github/workflows/deploy-to-hf.yml'))\""
100
+ run_test "Backup workflow syntax" "python3 -c \"import yaml; yaml.safe_load(open('.github/workflows/backup-workflows.yml'))\""
101
+ run_test "Sync workflow syntax" "python3 -c \"import yaml; yaml.safe_load(open('.github/workflows/sync-knowledge.yml'))\""
102
+ fi
103
+ }
104
+
105
+ # Test database connectivity
106
+ test_database() {
107
+ log_info "Testing database connectivity..."
108
+
109
+ if [[ -n "${DB_POSTGRESDB_HOST:-}" && -n "${DB_POSTGRESDB_PASSWORD:-}" ]]; then
110
+ export PGPASSWORD="$DB_POSTGRESDB_PASSWORD"
111
+
112
+ run_test "Database connection" "pg_isready -h '$DB_POSTGRESDB_HOST' -p '${DB_POSTGRESDB_PORT:-5432}' -U '$DB_POSTGRESDB_USER'"
113
+ run_test "Database access" "psql -h '$DB_POSTGRESDB_HOST' -p '${DB_POSTGRESDB_PORT:-5432}' -U '$DB_POSTGRESDB_USER' -d '$DB_POSTGRESDB_DATABASE' -c 'SELECT 1;'"
114
+ run_test "pgvector extension" "psql -h '$DB_POSTGRESDB_HOST' -p '${DB_POSTGRESDB_PORT:-5432}' -U '$DB_POSTGRESDB_USER' -d '$DB_POSTGRESDB_DATABASE' -c 'SELECT * FROM pg_extension WHERE extname = \\'vector\\';'"
115
+ else
116
+ log_warn "Database credentials not available - skipping connectivity tests"
117
+ fi
118
+ }
119
+
120
+ # Test knowledge base
121
+ test_knowledge() {
122
+ log_info "Testing knowledge base configuration..."
123
+
124
+ run_test "Knowledge directories exist" "[[ -d knowledge/n8n && -d knowledge/videos-e-animacoes && -d knowledge/midjourney-prompt ]]"
125
+
126
+ # Test Python dependencies for embedding generation
127
+ if command -v python3 > /dev/null; then
128
+ run_test "Python available" "python3 --version"
129
+ run_test "Required Python packages" "python3 -c 'import sentence_transformers, json, hashlib'"
130
+ fi
131
+ }
132
+
133
+ # Integration tests
134
+ test_integration() {
135
+ log_info "Running integration tests..."
136
+
137
+ # Test if we can start services locally
138
+ if run_test "Start services locally" "docker-compose -f docker/docker-compose.yml up -d --build"; then
139
+ sleep 30
140
+
141
+ run_test "n8n health endpoint" "curl -f http://localhost:7860/healthz"
142
+ run_test "Vector store endpoint" "curl -f http://localhost:8000/api/v1/heartbeat"
143
+
144
+ # Cleanup
145
+ docker-compose -f docker/docker-compose.yml down > /dev/null 2>&1
146
+ else
147
+ log_error "Failed to start services - skipping integration tests"
148
+ fi
149
+ }
150
+
151
+ # Main test execution
152
+ main() {
153
+ log_info "🧪 Starting n8n Infrastructure Test Suite"
154
+ echo "================================================"
155
+
156
+ cd "$PROJECT_ROOT"
157
+
158
+ # Run all test categories
159
+ test_docker
160
+ test_environment
161
+ test_scripts
162
+ test_github_actions
163
+ test_database
164
+ test_knowledge
165
+
166
+ # Skip integration tests if Docker unavailable
167
+ if docker --version > /dev/null 2>&1; then
168
+ test_integration
169
+ else
170
+ log_warn "Docker not available - skipping integration tests"
171
+ fi
172
+
173
+ # Test summary
174
+ echo ""
175
+ echo "================================================"
176
+ log_info "🎯 Test Results Summary"
177
+ echo " Total Tests: $TESTS_TOTAL"
178
+ echo " Passed: $TESTS_PASSED"
179
+ echo " Failed: $TESTS_FAILED"
180
+
181
+ if [[ $TESTS_FAILED -eq 0 ]]; then
182
+ echo " Status: ✅ ALL TESTS PASSED"
183
+ exit 0
184
+ else
185
+ echo " Status: ❌ SOME TESTS FAILED"
186
+ exit 1
187
+ fi
188
+ }
189
+
190
+ main "$@"