Managing Index Size

Exclude unnecessary files and folders from repo to index faster!

What is Indexable Size?

Indexable size is size of all code files, excluding following from the folder:

• Directory/File based filtering

  • logs, node_modules, dist, target, bin, package-lock.json, data.json, build, .gradle, .idea, gradle, extension.js, vendor.js, ngsw.json, polyfills.js, ngsw-worker.js, runtime.js, runtime-main.js, service-worker.js, bundle.js, bundle.css

• Extension based filtering

  • bin, exe, dll, log, aac, avif, bmp, cda, gif ,mp3, mp4, mpeg, weba, webm, webp, oga, ogv, png, jpeg, jpg, bmp, wpa, tif, tiff, svg, ico, wav, mov, avi, doc, docx, ppt, pptx, xls, xlsx, ods, odp, odt, pdf, epub, rar, tar, zip, vsix, 7z, bz, bz2, gzip, jar, war, gz, tgz, woff, woff2, eot, ttf, map, apk, app, ipa, lock, tmp, logs, gmo, pt

• Hidden files are filtered i.e., files starting with "."

• All Empty files are filtered.

• All Binary files are also filtered.

Is there any limit on repository size?

For workspaces that have upgraded to Bito's 10X Developer Plan, we have set the indexable size limit to 120MB per repo. However, once we launch the "AI that Understands Your Code" feature for our Forever Free users, they will be restricted to repositories with an indexable size limit of 10MB.

Learn more about indexable size above and see which files and folders are excluded by default.

You can reduce your repo's indexable size by excluding certain files and folders from indexing using .bitoignore file and remain within the limit.

What if a repo hits 120MB limit?

If a repo hits 120MB limit, then the below error message will be displayed in the "Manage repos" tab and the repo's index status will be changed to "Not Indexed".

Sorry, we don’t currently support repos of this size. Please use .bitoignore to reduce the size of the repo you want Bito to index.

To fix this issue, follow our instructions regarding how to use .bitoignore file and reduce your repo's size and bring it under the max limit of 120MB.

After that, you must delete the index and then restart the indexing by clicking on the "Start Indexing" button shown for the repo folder. You can also follow our step-by-step guides to Start Indexing in Visual Studio Code and Start Indexing in JetBrains IDEs.

What is .bitoignore and how to use it?

A .bitoignore file is a plain text file where each line contains a pattern or rules that tells Bito which files or directories to ignore and not index. In other words, it's a way to reduce your repo's indexable size. You can also see which files/folders are excluded by default.

There are two ways to use .bitoignore file:

1. Create a .bitoignore file inside the folder where indexes created by Bito are stored. e.g. <user-home-directory>/.bito/localcodesearch/.bitoignore

   • On Windows, this path will be something like: C:\Users\<your username>\.bito\localcodesearch\.bitoignore

   • Note: The custom ignore rules you set in this .bitoignore file will be applied to all the repositories where you have enabled indexing.

2. Create a .bitoignore file inside your repository's root folder.

If a .gitignore file is available in your repo then Bito will also use that to ignore files & folders from indexing process. Both .bitoignore and .gitignore files can work together without any issues.

At present, Bito considers only those .gitignore files that are placed in the project root directory and .bitoignore files that are placed either in <user-home-directory\.bito\localcodesearch> or <project-root-directory>

Changes to the .bitoignore file are taken into account at the beginning of the indexing process, not during or after the indexing itself.

Therefore, to implement changes made to the .bitoignore file, you'll need to delete the index and then restart the indexing by clicking on the "Start Indexing" button shown for the repo folder. You can also follow our step-by-step guides to Start Indexing in Visual Studio Code and Start Indexing in JetBrains IDEs.

Please note that any changes to the .bitoignore or .gitignore file will take a minimum of 3 to 5 minutes to trigger new indexing.

Common .bitoignore Patterns

Understanding these patterns/rules is crucial for effectively managing the files and directories that Bito indexes and excludes in your projects.

Sample Rule	Description
# this is a comment.	Any line that starts with a # symbol is considered as a comment and will not be processed.
*	(Wildcard character) Ignores all files
**	(Wildcard character) Match any number of directories.
todo.txt	Ignores a specific file named todo.txt
*.txt	Ignores all files ending with .txt
*.*	Ignores all files with any extension.
Engine/ or Engine/**	Ignores all files in the Engine directory and their subdirectories (contents).
subdirectory1/example.html	Ignore the file named example.html, specifically located in the directory named subdirectory1.
!contacts.txt	(Negation Rule) Explicitly tracks contacts.txt, even if all .txt files are ignored.
!Engine/Batch/Builds	(Negation Rule) Tracks the Builds directory inside Engine/Batch, overriding a broader exclusion.
!Engine/Batch/Builds/**	(Negation Rule) Tracks the Builds directory and all of its subdirectories inside Engine/Batch, overriding a broader exclusion.
!.java	(Negation Rule) Ensures that all .java files are included, overriding any previous ignore rules that might apply to them.
!subdirectory1/*.txt	(Negation Rule) Track files with the .txt extension located specifically in the subdirectory1 directory, even if other rules might otherwise ignore .txt files.
BitoUtil?.java	The ? (question mark) matches any single character in a filename or directory name.

Negation ! (exclamation mark)

When a pattern starts with ! it negates the pattern, meaning it explicitly includes files or directories that would otherwise be ignored. For example, have a look at this sample .bitoignore file:

.bitoignore

Engine/**
!Engine/Build/BatchFiles/**

Here !Engine/Build/BatchFiles/** pattern includes all files in the Engine/Build/BatchFiles directory and its subdirectories, even though Engine/** pattern would ignore them.

Avoid Ambiguous Patterns: Negation patterns can become confusing when they potentially match multiple files. Target specific files or folders rather than using wildcards in negation patterns.

For example, it is better to use patterns like !Engine/Build/BatchFiles/script.bat instead of !Engine/Build/BatchFiles/**

.bitoignore Examples

Exclude Files/Folders

# Ignore specific file named "config.ini"
config.ini

# Ignore all files with a '.bak' extension
*.bak

# Ignore all files with a '.kunal' extension
*.kunal

# Exclude directories
backup
temp/
dist/vendor

Exclude Everything Except Specific Files

To exempt a file, ensure that the negation pattern ! appears afterward, thereby overriding any broader exclusions.

# Ignore all files except C++, header and python files
*
!*.cpp
!*.h
!*.py