Refactoring and Improvements

.github/workflows/main.yml

@@ -56,6 +56,7 @@ jobs:
         run: |
           poetry install --no-interaction --no-root
           poetry install --with dev 
+          poetry install --extras "spacy"
 
       - name: Run Pytest with coverage
         run: |

Makefile

@@ -10,6 +10,8 @@ test:
 coverage:
 	poetry run pytest -vvv --cov
 
+format:
+	poetry run ruff format .
 
 .PHONY: test coverage 
 

README.md

@@ -7,17 +7,34 @@ ShadowData is a Python library designed to simplify the processing and handling
 
 ## Features (The project is under development)
 
-- Data anonymization.
-- Encryption and decryption of sensitive data.
-- Data masking for privacy-preserving data handling.
-- Easy integration with existing Python data processing pipelines.
-- Compliance with GDPR, LGPD and other data protection regulations.
+- Data anonymization. (Work in progress)
+- PII - Personal Identified Information detection using Natual Language Processing (Work in progress) 
+- Encryption and decryption of sensitive data. (Development has not started yet)
+- Data masking for privacy-preserving data handling. (Work in progress)
+- Compliance with GDPR, LGPD and other data protection regulations. (Work in progress)
 
-## Installing
-`pip install shadow_data`
+## Installation Instructions
+
+```bash
+pip install shadow_data
+```
+* Installs only the core library, without the spaCy dependency.
+
+```bash 
+pip install shadowdata[spacy]
+```
+* Installs spaCy automatically, based on your platform.
+
+By default, ShadowData will automatically download the necessary language model if it’s not already installed. However, if you’d prefer to install it manually, use the following command:
+```bash
+python -m spacy download en_core_web_trf
+```
+Make sure to run this command within your project’s virtual environment.
+
+[Check spaCy's documentation to know more about the Language Models.](https://spacy.io/models)
 
 ## Usage
-Examples soon
+There are some usage examples at the [examples](examples) directory
 
 ## Contributing
 

examples/anonymization.md

@@ -0,0 +1,20 @@
+# Simple text replacement and IPV4 anonymization
+
+```python
+from shadow_data.anonymization import TextProcessor, Ipv4Anonymization
+
+original_content = 'Little John likes to play with the staff.'
+anonymized_content = TextProcessor.replace_text('Little John', 'XXXXXX XXXX', original_content)
+print(f'Original content: {original_content} | Anonymized content: {anonymized_content}')
+
+original_ip = '200.189.45.128'
+anonymized_content = Ipv4Anonymization.anonymize_ipv4(original_ip)
+print(f'Original IP: {original_ip} | Anonymized IP: {anonymized_content}')
+```
+
+### Results:
+
+```plain
+Original content: Little John likes to play with the staff. | Anonymized content: XXXXXX XXXX likes to play with the staff.
+Original IP: 200.189.45.128 | Anonymized IP: 200.X.X.X
+```

examples/i10n_brazil.md

@@ -0,0 +1,24 @@
+# Localized anonymization for Brazilian individual (CPF) and corporate (CNPJ) tax IDs.
+
+```python
+from shadow_data.l10n.brazil import IdentifierAnonymizer
+
+# Anonymize CPF
+
+cpf = '806.846.761-09'
+cpf_anonymized = IdentifierAnonymizer(cpf)
+cpf_anonymized.anonymize()
+print(f'Original CPF: {cpf} : Anonymized CPF: {cpf_anonymized.cleaned_content}')
+
+cnpj = '26.283.050/0001-17'
+cnpj_anonymized = IdentifierAnonymizer(cnpj)
+cnpj_anonymized.anonymize()
+print(f'Original CNPJ: {cnpj} : Anonymized CPF: {cnpj_anonymized.cleaned_content}')
+```
+
+### Results
+
+```plain
+Original CPF: 806.846.761-09 : Anonymized CPF: 80*********
+Original CNPJ: 26.283.050/0001-17 : Anonymized CPF: 26************
+```

examples/i10n_us.md

@@ -0,0 +1,17 @@
+# Localized anonymization for U.S. Social Security Numbers.
+
+```python
+from shadow_data.l10n.usa import IdentifierAnonymizer
+
+text_content_with_ssn = "Billy's SSN is 479-92-5042. Please make sure it's anonymized."
+anonymizer = IdentifierAnonymizer(text_content_with_ssn)
+anonymizer.anonymize()
+
+print(f'Original: {text_content_with_ssn} | Anonymized: {anonymizer.cleaned_content}')
+```
+
+### Results
+
+```plain
+Original: Billy's SSN is 479-92-5042. Please make sure it's anonymized. | Anonymized: Billy's SSN is XXX-XX-5042. Please make sure it's anonymized.
+```

examples/pii_nlp.md

@@ -0,0 +1,24 @@
+# Personally Identifiable Information with Natural Language Processing
+
+PII, or Personally Identifiable Information, refers to any data that can be used to identify a specific individual. This includes names, addresses, phone numbers, social security numbers, email addresses, and other personal details that can be linked to a person. Protecting PII is essential for maintaining privacy and ensuring data security.
+
+spaCy is an open-source library for advanced natural language processing (NLP) in Python. It provides fast and efficient tools for tasks like tokenization, part-of-speech tagging, named entity recognition, and more, with support for multiple languages.
+
+This library primarily uses spaCy to process text and identify personally identifiable information (PII).
+
+**_The code sample below is in Portuguese because the results were better than those from the tests I conducted in English._**
+
+```python 
+from shadow_data.pii.enums import ModelLang, ModelCore, ModelSize
+from shadow_data.pii.spacy import SensitiveData
+
+content = 'João Silva mora na Rua das Acácias, 123, Apartamento 5B, São Paulo, SP, 01310-000. Ele trabalha como Gerente de Projetos na TechNova Soluções, uma empresa de tecnologia em crescimento localizada na Avenida Paulista, 987, São Paulo, SP, 01311-200. João está na empresa há três anos, onde lidera uma equipe de desenvolvedores e designers, garantindo que os projetos sejam entregues no prazo e atendam às expectativas dos clientes. Você pode contatá-lo pelo telefone (11) 91234-5678 ou pelo e-mail joao.silva@technova.com. João é muito respeitado por suas habilidades de liderança e resolução de problemas dentro da empresa.'
+instance = SensitiveData()
+sensitive_data = instance.identify_sensitive_data(ModelLang.PORTUGUESE, ModelCore.NEWS, ModelSize.LARGE, content)
+print(sensitive_data)
+```
+
+### Results:
+```plain
+[('João Silva', 'PER'), ('Rua das Acácias', 'LOC'), ('Apartamento 5B', 'LOC'), ('São Paulo', 'LOC'), ('SP', 'LOC'), ('Gerente de Projetos', 'MISC'), ('TechNova Soluções', 'LOC'), ('Avenida Paulista', 'LOC'), ('São Paulo', 'LOC'), ('SP', 'LOC'), ('João', 'PER'), ('joao.silva@technova.com', 'LOC'), ('João', 'PER')]
+```