flatisfy package

Subpackages

• flatisfy.data_files package
  • Module contents
• flatisfy.database package
  • Submodules
  • flatisfy.database.base module
  • flatisfy.database.types module
  • flatisfy.database.whooshalchemy module
    • WhooshAlchemy
  • Module contents
• flatisfy.filters package
  • Submodules
  • flatisfy.filters.cache module
  • flatisfy.filters.duplicates module
  • flatisfy.filters.metadata module
  • Module contents
• flatisfy.models package
  • Submodules
  • flatisfy.models.flat module
  • flatisfy.models.postal_code module
  • flatisfy.models.public_transport module
  • Module contents
• flatisfy.web package
  • Subpackages
    • flatisfy.web.routes package
      • Submodules
      • flatisfy.web.routes.api module
      • Module contents
  • Submodules
  • flatisfy.web.app module
  • flatisfy.web.configplugin module
  • flatisfy.web.dbplugin module
  • Module contents

Submodules

flatisfy.cmds module

Main commands available for flatisfy.

flatisfy.cmds.filter_fetched_flats(config, fetched_flats, fetch_details=True)

Filter the available flats list. Then, filter it according to criteria.

Parameters:

• config – A config dict.
• fetch_details – Whether additional details should be fetched between the two passes.
• fetched_flats – The initial dict mapping constraints to the list of fetched flat objects to filter.

Returns:

A dict mapping constraints to a dict mapping flat status and list of flat objects.

flatisfy.cmds.filter_flats_list(config, constraint_name, flats_list, fetch_details=True)

Filter the available flats list. Then, filter it according to criteria.

Parameters:

• config – A config dict.
• constraint_name – The constraint name that the flats_list should satisfy.
• fetch_details – Whether additional details should be fetched between the two passes.
• flats_list – The initial list of flat objects to filter.

Returns:

A dict mapping flat status and list of flat objects.

flatisfy.cmds.import_and_filter(config, load_from_db=False)

Fetch the available flats list. Then, filter it according to criteria. Finally, store it in the database.

Parameters:

• config – A config dict.
• load_from_db – Whether to load flats from database or fetch them using WebOOB.

Returns:

None.

flatisfy.cmds.purge_db(config)

Purge the database.

Parameters:config – A config dict. Returns:None

flatisfy.cmds.serve(config)

Serve the web app.

Parameters:config – A config dict. Returns:None, long-running process.

flatisfy.config module

This module handles the configuration management for Flatisfy.

It loads the default configuration, then overloads it with the provided config file and then overloads it with command-line options.

flatisfy.config.init_config(output=None)

Initialize an empty configuration file.

Parameters:output – File to output content to. Defaults to stdin.

flatisfy.config.load_config(args=None, check_with_data=True)

Load the configuration from file.

Parameters:

• args – An argparse args structure.
• check_with_data – Whether we should use the available OpenData to check the config values. Defaults to True.

Returns:

The loaded config dict.

flatisfy.config.validate_config(config, check_with_data)

Check that the config passed as argument is a valid configuration.

Parameters:

• config – A config dictionary to fetch.
• check_with_data – Whether we should use the available OpenData to check the config values.

Returns:

True if the configuration is valid, False otherwise.

flatisfy.constants module

Constants used across the app.

class flatisfy.constants.TimeToModes

Bases: enum.Enum

An enumeration.

BIKE = 2

CAR = 3

PUBLIC_TRANSPORT = -1

WALK = 1

flatisfy.data module

This module contains all the code related to building necessary data files from the source opendata files.

flatisfy.data.preprocess_data(config, force=False)

Ensures that all the necessary data have been inserted in db from the raw opendata files.

Params config:A config dictionary. Params force:Whether to force rebuild or not. Return bool:Whether data have been built or not.

flatisfy.email module

Email notifications.

flatisfy.email.send_email(server, port, subject, _from, _to, txt, html, username=None, password=None)

Send an email

Parameters:

• server – SMTP server to use.
• port – SMTP port to use.
• subject – Subject of the email to send.
• _from – Email address of the sender.
• _to – List of email addresses of the receivers.
• txt – Text version of the message.
• html – HTML version of the message.

flatisfy.email.send_notification(config, flats)

Send an email notification about new available flats.

Parameters:

• config – A config dict.
• flats – List of flats to include in the notification.

flatisfy.exceptions module

This module contains all the exceptions definitions for the Flatisfy-specific exceptions.

exception flatisfy.exceptions.DataBuildError

Bases: Exception

Error occurring on building a data file.

flatisfy.fetch module

This module contains all the code related to fetching and loading flats lists.

class flatisfy.fetch.WebOOBProxy(config)

Bases: object

Wrapper around WebOOB WebNip class, to fetch housing posts without having to spawn a subprocess.

build_queries(constraints_dict)

Build WebOOB weboob.capabilities.housing.Query objects from the constraints defined in the configuration. Each query has at most 3 cities, to comply with housing websites limitations.

Parameters:constraints_dict – A dictionary of constraints, as defined in the config. Returns:A list of WebOOB weboob.capabilities.housing.Query objects. Returns None if an error occurred.

info(full_flat_id, store_personal_data=False)

Get information (details) about an housing post.

Parameters:

• full_flat_id – A WebOOB housing post id, in complete form (ID@BACKEND)
• store_personal_data – Whether personal data should be fetched from housing posts (phone number etc).

Returns:

The details in JSON.

query(query, max_entries=None, store_personal_data=False)

Fetch the housings posts matching a given WebOOB query.

Parameters:

• query – A WebOOB weboob.capabilities.housing.Query` object.
• max_entries – Maximum number of entries to fetch.
• store_personal_data – Whether personal data should be fetched from housing posts (phone number etc).

Returns:

The matching housing posts, dumped as a list of JSON objects.

static restore_decimal_fields(flat)

Parse fields expected to be in Decimal type to float. They were dumped as str in the JSON dump process.

Parameters:flat – A flat dict. Returns:A flat dict with Decimal fields converted to float.

static version()

Get WebOOB version.

Returns:The installed WebOOB version.

flatisfy.fetch.fetch_details(config, flat_id)

Fetch the additional details for a flat using Flatboob / WebOOB.

Parameters:

• config – A config dict.
• flat_id – ID of the flat to fetch details for.

Returns:

A flat dict with all the available data.

flatisfy.fetch.fetch_flats(config)

Fetch the available flats using the Flatboob / WebOOB config.

Parameters:config – A config dict. Returns:A dict mapping constraint in config to all available matching flats.

flatisfy.fetch.load_flats_from_db(config)

Load flats from database.

Parameters:config – A config dict. Returns:A dict mapping constraint in config to all available matching flats.

flatisfy.fetch.load_flats_from_file(json_file, config)

Load a dumped flats list from JSON file.

Parameters:json_file – The file to load housings list from. Returns:A dict mapping constraint in config to all available matching flats.

Note

As we do not know which constraint is met by a given flat, all the flats are returned for any available constraint, and they will be filtered out afterwards.

flatisfy.tests module

This module contains unit testing functions.

class flatisfy.tests.LocalImageCache(max_items=200, storage_dir=None)

Bases: flatisfy.filters.cache.ImageCache

A local cache for images, stored in memory.

static on_miss(path)

Helper to actually retrieve photos if not already cached.

class flatisfy.tests.TestDuplicates(*args, **kwargs)

Bases: unittest.case.TestCase

Checks duplicates detection.

DUPLICATES_MIN_SCORE_WITHOUT_PHOTOS = 8

DUPLICATES_MIN_SCORE_WITH_PHOTOS = 15

HASH_THRESHOLD = 10

static generate_fake_flat()

Generates a fake flat post.

static load_files(file1, file2)

Load two files

Returns:A dict with two flats

test_different_areas()

Two flats with different areas should not be detected as duplicates.

test_different_areas_decimals()

Two flats which areas integers are equal but decimals are present and different should not be detected as duplicates.

test_different_phones()

Two flats with different phone numbers should not be detected as duplicates.

test_different_prices()

Two flats with different prices should not be detected as duplicates.

test_different_rooms()

Two flats with different rooms quantity should not be detected as duplicates.

test_duplicates()

Two identical flats should be detected as duplicates.

test_real_duplicates()

Two flats with same price, area and rooms quantity should be detected as duplicates.

class flatisfy.tests.TestImageCache(*args, **kwargs)

Bases: unittest.case.TestCase

Checks image cache is working as expected.

test_invalid_data()

Check that it returns nothing on an invalid data.

test_invalid_url()

Check that it returns nothing on an invalid URL.

class flatisfy.tests.TestPhoneNumbers(methodName='runTest')

Bases: unittest.case.TestCase

Checks phone numbers normalizations.

test_dots_separators()

Checks phone numbers with dots.

test_prefix()

Checks phone numbers with international prefixes.

test_spaces_separators()

Checks phone numbers with spaces.

class flatisfy.tests.TestPhotos(*args, **kwargs)

Bases: unittest.case.TestCase

HASH_THRESHOLD = 10

test_different_photos()

Compares two different photos.

test_matching_cropped_photos()

Compares two matching photos with one being cropped.

test_matching_photos()

Compares two matching photos with different size and source.

test_same_photo_twice()

Compares a photo against itself.

class flatisfy.tests.TestTexts(methodName='runTest')

Bases: unittest.case.TestCase

Checks string normalizations.

test_accents()

Checks accents are replaced.

test_multiple_whitespaces()

Checks whitespaces are collapsed.

test_roman_numbers()

Checks roman numbers replacement.

test_roman_numbers_in_text()

Checks conversion of roman numbers to arabic ones in string normalization.

test_whitespace_trim()

Checks that trailing and beginning whitespaces are trimmed.

flatisfy.tests.run()

Run all the tests

flatisfy.tools module

This module contains basic utility functions, such as pretty printing of JSON output, checking that a value is within a given interval etc.

class flatisfy.tools.DateAwareJSONEncoder(*, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None)

Bases: json.encoder.JSONEncoder

Extend the default JSON encoder to serialize datetimes to iso strings.

default(o)

Implement this method in a subclass such that it returns a serializable object for o, or calls the base implementation (to raise a TypeError).

For example, to support arbitrary iterators, you could implement default like this:

def default(self, o):
    try:
        iterable = iter(o)
    except TypeError:
        pass
    else:
        return list(iterable)
    # Let the base class default method raise the TypeError
    return JSONEncoder.default(self, o)

flatisfy.tools.batch(iterable, size)

Get items from a sequence a batch at a time.

Parameters:

• iterable – The iterable to get the items from.
• size – The size of the batches.

Returns:

A new iterable.

flatisfy.tools.convert_arabic_to_roman(arabic)

Convert an arabic literal to a roman one. Limits to 39, which is a rough estimate for a maximum for using roman notations in daily life.

..note::

Based on https://gist.github.com/riverrun/ac91218bb1678b857c12.

Parameters:arabic – An arabic number, as string. Returns:The corresponding roman one, as string.

flatisfy.tools.convert_arabic_to_roman_in_text(text)

Convert roman literals to arabic one in a text.

Parameters:text – Some text to convert roman literals from. Returns:The corresponding text with roman literals converted to arabic.

flatisfy.tools.distance(gps1, gps2)

Compute the distance between two tuples of latitude and longitude.

Parameters:

• gps1 – First tuple of (latitude, longitude).
• gps2 – Second tuple of (latitude, longitude).

Returns:

The distance in meters.

Example:

>>> int(distance([48.86786647303717, 2.19368117495212],                          [48.95314107920405, 2.3368043817358464]))
14117

flatisfy.tools.get_travel_time_between(latlng_from, latlng_to, mode, config)

Query the Navitia API to get the travel time between two points identified by their latitude and longitude.

Parameters:

• latlng_from – A tuple of (latitude, longitude) for the starting point.
• latlng_to – A tuple of (latitude, longitude) for the destination.
• mode – A TimeToMode enum value for the mode of transportation to use.

Returns:

A dict of the travel time in seconds and sections of the journey with GeoJSON paths. Returns None if it could not fetch it.

Note

Uses the Navitia API. Requires a navitia_api_key field to be filled-in in the config.

flatisfy.tools.hash_dict(func)

Decorator to use on functions accepting dict parameters, to transform them into immutable dicts and be able to use lru_cache.

From https://stackoverflow.com/a/44776960.

flatisfy.tools.is_within_interval(value, min_value=None, max_value=None)

Check whether a variable is within a given interval. Assumes the value is always ok with respect to a None bound. If the value is None, it is always within the bounds.

Parameters:

• value – The value to check. Can be None.
• min_value – The lower bound.
• max_value – The upper bound.

Returns:

True if the value is None. True or False whether the value is within the given interval or not.

Note

A value is always within a None bound.

Example:

>>> is_within_interval(None)
True
>>> is_within_interval(None, 0, 10)
True
>>> is_within_interval(2, None, None)
True
>>> is_within_interval(2, None, 3)
True
>>> is_within_interval(2, 1, None)
True
>>> is_within_interval(2, 1, 3)
True
>>> is_within_interval(2, 4, 7)
False
>>> is_within_interval(2, 4, 1)
False

flatisfy.tools.merge_dicts(*args)

Merge the two flats passed as argument in a single flat dict object.

flatisfy.tools.next_weekday(d, weekday)

Find datetime object for next given weekday.

From https://stackoverflow.com/questions/6558535/find-the-date-for-the-first-monday-after-a-given-a-date.

Parameters:

• d – Datetime to search from.
• weekday – Weekday (0 for Monday, etc)

Returns:

The datetime object for the next given weekday.

flatisfy.tools.normalize_string(string, lowercase=True, convert_arabic_numerals=True)

Normalize the given string for matching.

Example:

>>> normalize_string("tétéà 14ème-XIV,  foobar")
'tetea XIVeme xiv, foobar'

>>> normalize_string("tétéà 14ème-XIV,  foobar", False)
'tetea 14eme xiv, foobar'

Parameters:

• string – The string to normalize.
• lowercase – Whether to convert string to lowercase or not. Defaults to True.
• convert_arabic_numerals – Whether to convert arabic numerals to roman ones. Defaults to True.

Returns:

The normalized string.

flatisfy.tools.pretty_json(data)

Pretty JSON output.

Parameters:

data – The data to dump as pretty JSON.

Returns:

The pretty printed JSON dump.

Example:

>>> print(pretty_json({"toto": "ok", "foo": "bar"}))
{
    "foo": "bar",
    "toto": "ok"
}

flatisfy.tools.sort_list_of_dicts_by(flats_list, key)

Sort a list of dicts according to a given field common to all the dicts.

Parameters:

• flats_list – List of dicts to sort.
• key – The key of the dict items to sort on.

Returns:

A sorted list.

Example:

>>> sort_list_of_dicts_by([{1: 2}, {1: 1}], 1)
[{1: 1}, {1: 2}]

flatisfy.tools.timeit(func)

A decorator that logs how much time was spent in the function.

flatisfy.tools.uniqify(some_list)

Filter out duplicates from a given list.

Example:

>>> uniqify([1, 2, 2, 3])
[1, 2, 3]

Module contents

Flatisfy is a tool to help you find a new housing based on some criteria.