PhoneInfoga#995 #2107

g4ze · 2024-01-31T15:53:44Z

closes #995

Description

Add phoneinfoga_scan analyzer

Type of change

Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue).
New feature (non-breaking change which adds functionality).
Breaking change (fix or feature that would cause existing functionality to not work as expected).

Checklist

Important Rules

If you miss to compile the Checklist properly, your PR won't be reviewed by the maintainers.
If your changes decrease the overall tests coverage (you will know after the Codecov CI job is done), you should add the required tests to fix the problem
Everytime you make changes to the PR and you think the work is done, you should explicitly ask for a review. After being reviewed and received a "change request", you should explicitly ask for a review again once you have made the requested changes.

raw json:
{"result":{"valid":true,"number":"33679368229","local_format":"0679368229","international_format":"+33679368229","country_prefix":"+33","country_code":"FR","country_name":"France","location":"","carrier":"Orange France SA","line_type":"mobile"}}

g4ze · 2024-01-31T16:03:51Z

I still need to update docs and make migrations

…into PhoneInfoga#995

g4ze · 2024-01-31T18:34:18Z

Docs done

g4ze · 2024-01-31T18:34:30Z

Hello! @mlodic , this took me sometime to get done. The test is failing here. It pertains to the migration file. I used the dumpplugin comand to create the file. I've crossed checked it with the way previous migrations are written. Still can't seem to find out why this is happening.

mlodic · 2024-02-01T17:26:39Z

integrations/phoneinfoga/compose.yml

+      environment:
+      - GIN_MODE=release


reference to this?

this should run the GIN framework used in release mode. It works fine even without it.

mlodic · 2024-02-01T17:28:13Z

integrations/phoneinfoga/compose-tests.yml

+version: '3.7'
+
+services:
+    phoneinfoga:
+      container_name: phoneinfoga
+      restart: unless-stopped
+      image: sundowndev/phoneinfoga:latest
+      espose:
+        - "5000"
+      env_file:
+      - env_file_integrations
+      environment:
+      - GIN_MODE=release


copy/pastes are not required because the compose-tests.yml is an override of the compose.yml. Please use other docker analyzers as a reference

so should this file even contain anything as we are using an external service; would it make sense to have an additional dockerfile that clones the repo and builds it for the compose-tests.yml?

you are right, we could basically leave this almost empty.

should I just mention version: 3.8 in the test yml and rest empty?

integrations/phoneinfoga/compose.yml

docs/source/Advanced-Usage.md

api_app/analyzers_manager/migrations/0064_analyzer_config_phoneinfoga.py

mlodic · 2024-02-02T08:20:37Z

integrations/phoneinfoga/compose.yml

@@ -0,0 +1,16 @@
+version: '3.7'


use version 3.8 if you can, like we do in all other docker compose files in the project

mlodic · 2024-02-02T13:59:02Z

the migration error is here:

  File "/opt/deploy/intel_owl/api_app/analyzers_manager/migrations/0064_analyzer_config_phoneinfoga.py", line 118, in migrate
    _create_object(PluginConfig, value)

I think that there is some problem with the IDs

django.db.utils.IntegrityError: duplicate key value violates unique constraint "api_app_parameter_name_python_module_id_85f61e18_uniq"

I think that could be the error, I asking to @0ssigeno a second opinion

0ssigeno · 2024-02-05T08:05:36Z

Checking

…into PhoneInfoga#995

g4ze · 2024-02-05T12:19:06Z

uhoh, i did it again; overlooked a checkpoint. Made required migrations(agn)

g4ze · 2024-02-07T18:43:14Z

it worked lol. @mlodic any other changes?

mlodic

you can ignore codecov errors, it's a problem of their server. We disabled it.

Considering that I merged Validin PR, could you please adjust the migration here again :P too many PR :P

api_app/analyzers_manager/observable_analyzers/phoneinfoga_scan.py

mlodic · 2024-02-08T10:03:31Z

api_app/playbooks_manager/migrations/0027_add_phoneinfoga_scan_to_free_to_use_analyzers.py

+# This file is a part of IntelOwl https://github.com/intelowlproject/IntelOwl
+# See the file 'LICENSE' for copying permission.
+
+
+from django.db import migrations
+
+
+def migrate(apps, schema_editor):
+    playbook_config = apps.get_model("playbooks_manager", "PlaybookConfig")
+    AnalyzerConfig = apps.get_model("analyzers_manager", "AnalyzerConfig")
+    pc = playbook_config.objects.get(name="FREE_TO_USE_ANALYZERS")
+    pc.analyzers.add(AnalyzerConfig.objects.get(name="Phoneinfoga").id)
+    pc.full_clean()
+    pc.save()
+
+
+def reverse_migrate(apps, schema_editor):
+    playbook_config = apps.get_model("playbooks_manager", "PlaybookConfig")
+    AnalyzerConfig = apps.get_model("analyzers_manager", "AnalyzerConfig")
+    pc = playbook_config.objects.get(name="FREE_TO_USE_ANALYZERS")
+    pc.analyzers.remove(AnalyzerConfig.objects.get(name="Phoneinfoga").id)
+    pc.full_clean()
+    pc.save()
+
+
+class Migration(migrations.Migration):
+    dependencies = [
+        ("playbooks_manager", "0026_add_zippy_scan_to_free_to_use"),
+        ("analyzers_manager", "0065_analyzer_config_phoneinfoga"),
+    ]
+
+    operations = [
+        migrations.RunPython(migrate, reverse_migrate),
+    ]


This is not required because this analyzer is "optional" and it requires to run an additional container.
To avoid to have it providing errors in the base installation, that is the most commonly used, I would not add it in the default playbook. Please remove.

noted. Shouldn't we also mention that such type of analyzers which are free but docker-based/optional can't go in default playbook in the docs or the PR checklist; to avoid such implementations in future.

sure it would make sense to explicitly say it in the PR template

just one more concern, about another docker based analyzer, cyberchef which is in the default playbook

mlodic · 2024-02-08T10:07:33Z

docker/env_file_integrations_template

+LOG_LEVEL=INFO
+# For phoneinfoga analyzer 
+# NUMVERIFY_API_KEY=


isn't there a way to leverage classic Analyzer Secrets for this API key? In that way, every user can configure it and use its own API key.
I don't like to have it configured here because this API key would be shared among all the users. This could be an unwanted behavior.

I'll look into that. Will have to figure out a workaround :'')

hi there! I took a look at their code base. The problem at hand involves the extraction of all the API keys from the from the integration .env file instead of directly using the key as authorization header when making calls. This is hard coded in their code base.
One work around could be making a new container everytime a new user selects the analyzer. This would have some infra implications but would allow us to have unique containers of the analyzer with its own env file for each individual user. (not a good approach tho)
I have raised an (issue)[https://github.com/sundowndev/phoneinfoga/issues/1393] for further enquiry.
In the meantime we could update the analyzer configs to take API keys from the user in the interface and implement that logic whenever they solve our problem, till then we might have to leverage integration env only.

thank you for the efforts. I would like to avoid to create complex things for this problem, I like to keep things simple if possible.
I suggest waiting for an answer from them. In case they do not answer within a week, we consider the problem unsolved and so we'll merge your actual PR even if there is that problem. In that case, we would need to specify more information about it in the docs, just to be explicit about this problem with the users.

for sure! In the meantime can you guide me to another issue i can take up?

this is a sightly different analyzer: #1103.
This is a really important source for this project

code quality Co-authored-by: Matteo Lodi <30625432+mlodic@users.noreply.github.com>

…into PhoneInfoga#995

mlodic · 2024-02-16T08:35:02Z

Just to keep a reference, they answered here sundowndev/phoneinfoga#1393)

We'll wait them to solve the issue regarding API keys than we'll finish this PR

g4ze · 2024-02-21T15:29:42Z

works using new version of phoneinfoga

g4ze · 2024-02-21T15:41:39Z

@mlodic kindly review.

mlodic · 2024-02-21T15:45:55Z

let's wait for the tests. If they pass we can merge the PR. Ty!

integrate phoneinfoga_scan analyzer

9c4c04f

g4ze changed the title ~~Phone infoga, closes #995~~ PhoneInfoga, closes #995 Jan 31, 2024

g4ze added 3 commits January 31, 2024 22:00

docs

73803b6

Merge branch 'develop' of https://github.com/intelowlproject/IntelOwl …

9662924

…into PhoneInfoga#995

migrations

4cf684a

docker based true

a186ba8

mlodic requested changes Feb 1, 2024

View reviewed changes

g4ze added 9 commits February 2, 2024 02:01

integrate phoneinfoga_scan analyzer

1a81261

docs

f264ad9

migrations

93a2986

docker based true

80d9015

typo

d2a6aba

gin mode remove; --no-client

dc05234

doc update: scanner_name desccription

abc6d3b

gin mode remove; --no-client

fe7b3ff

minor fix

0abc5c2

g4ze requested a review from mlodic February 1, 2024 20:52

mlodic reviewed Feb 2, 2024

View reviewed changes

compose version

52de367

g4ze requested review from mlodic February 2, 2024 08:44

g4ze added 4 commits February 4, 2024 15:15

test

c49597c

test

687815d

small fix

eb45821

minor fix

9645fa4

g4ze added 3 commits February 5, 2024 17:01

Merge branch 'develop' of https://github.com/intelowlproject/IntelOwl …

7faab97

…into PhoneInfoga#995

migration update

e79d51e

add_phoneinfoga_scan_to_free_to_use_analyzers

f9cd49c

g4ze added 2 commits February 5, 2024 19:10

migration fix

47bb8ec

typo

9d36533

g4ze changed the title ~~PhoneInfoga, closes #995~~ PhoneInfoga#995 Feb 8, 2024

mlodic requested changes Feb 8, 2024

View reviewed changes

g4ze and others added 2 commits February 8, 2024 16:01

code quality phoneinfoga_scan

93b8fbe

code quality Co-authored-by: Matteo Lodi <30625432+mlodic@users.noreply.github.com>

Merge branch 'develop' of https://github.com/intelowlproject/IntelOwl …

133654f

…into PhoneInfoga#995

g4ze mentioned this pull request Feb 8, 2024

workaround .env file sundowndev/phoneinfoga#1393

Closed

migration

b04981c

g4ze requested a review from mlodic February 8, 2024 14:02

linters

7885dab

g4ze added 7 commits February 21, 2024 19:43

dev pull

425c15d

phoneinfoga v2.11

22afed0

celery_reload linter

0effa10

migration

18ffb74

variable ficx

c80d3c8

just an empty string

5ae36d0

env file

fb35f62

docs

5fa6c8e

mlodic approved these changes Feb 21, 2024

View reviewed changes

mlodic merged commit 5d6c04a into intelowlproject:develop Feb 21, 2024
11 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PhoneInfoga#995 #2107

PhoneInfoga#995 #2107

g4ze commented Jan 31, 2024 •

edited

Loading

g4ze commented Jan 31, 2024

g4ze commented Jan 31, 2024

g4ze commented Jan 31, 2024

mlodic Feb 1, 2024

g4ze Feb 1, 2024

mlodic Feb 1, 2024

g4ze Feb 1, 2024

mlodic Feb 2, 2024

g4ze Feb 2, 2024

mlodic Feb 2, 2024

mlodic commented Feb 2, 2024 •

edited

Loading

0ssigeno commented Feb 5, 2024

g4ze commented Feb 5, 2024

g4ze commented Feb 7, 2024

mlodic left a comment

mlodic Feb 8, 2024

g4ze Feb 8, 2024

mlodic Feb 8, 2024

g4ze Feb 8, 2024

mlodic Feb 8, 2024

g4ze Feb 8, 2024

g4ze Feb 8, 2024

mlodic Feb 8, 2024

g4ze Feb 8, 2024

mlodic Feb 8, 2024

mlodic commented Feb 16, 2024

g4ze commented Feb 21, 2024

g4ze commented Feb 21, 2024

mlodic commented Feb 21, 2024

PhoneInfoga#995 #2107

PhoneInfoga#995 #2107

Conversation

g4ze commented Jan 31, 2024 • edited Loading

Description

Type of change

Checklist

Important Rules

g4ze commented Jan 31, 2024

g4ze commented Jan 31, 2024

g4ze commented Jan 31, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mlodic commented Feb 2, 2024 • edited Loading

0ssigeno commented Feb 5, 2024

g4ze commented Feb 5, 2024

g4ze commented Feb 7, 2024

mlodic left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mlodic commented Feb 16, 2024

g4ze commented Feb 21, 2024

g4ze commented Feb 21, 2024

mlodic commented Feb 21, 2024

g4ze commented Jan 31, 2024 •

edited

Loading

mlodic commented Feb 2, 2024 •

edited

Loading