diff --git a/README.md b/README.md
index 0eaded6..dc356f1 100644
--- a/README.md
+++ b/README.md
@@ -1,106 +1,205 @@
-
+
-**ArXiv Digest and Personalized Recommendations using Large Language Models.**
+# ArXiv Digest (Enhanced Edition)
-This repo aims to provide a better daily digest for newly published arXiv papers based on your own research interests and natural-language descriptions, using relevancy ratings from GPT.
+**Personalized arXiv Paper Recommendations with Multiple AI Models**
-You can try it out on [Hugging Face](https://huggingface.co/spaces/AutoLLM/ArxivDigest) using your own OpenAI API key.
-
-You can also create a daily subscription pipeline to email you the results.
+This repository provides an enhanced daily digest for newly published arXiv papers based on your research interests, leveraging multiple AI models including OpenAI GPT, Google Gemini, and Anthropic Claude to provide relevancy ratings, detailed analysis, and topic clustering.
## π Contents
-- [What this repo does](#π-what-this-repo-does)
- * [Examples](#some-examples)
-- [Usage](#π‘-usage)
- * [Running as a github action using SendGrid (Recommended)](#running-as-a-github-action-using-sendgrid-recommended)
- * [Running as a github action with SMTP credentials](#running-as-a-github-action-with-smtp-credentials)
- * [Running as a github action without emails](#running-as-a-github-action-without-emails)
- * [Running from the command line](#running-from-the-command-line)
- * [Running with a user interface](#running-with-a-user-interface)
-- [Roadmap](#β
-roadmap)
-- [Extending and Contributing](#π-extending-and-contributing)
+- [Features](#-features)
+- [Quick Start](#-quick-start)
+- [What This Repo Does](#-what-this-repo-does)
+- [Model Integrations](#-model-integrations)
+- [Design Paper Discovery](#-design-paper-discovery)
+- [Output Formats](#-output-formats)
+- [Setting Up and Usage](#-setting-up-and-usage)
+ * [Configuration](#configuration)
+ * [Running the Web Interface](#running-the-web-interface)
+ * [Running via GitHub Action](#running-via-github-action)
+ * [Running from Command Line](#running-from-command-line)
+- [API Usage Notes](#-api-usage-notes)
+- [Directory Structure](#-directory-structure)
+- [Roadmap](#-roadmap)
+- [Contributing](#-contributing)
-## π What this repo does
+## β¨ Features
-Staying up to date on [arXiv](https://arxiv.org) papers can take a considerable amount of time, with on the order of hundreds of new papers each day to filter through. There is an [official daily digest service](https://info.arxiv.org/help/subscribe.html), however large categories like [cs.AI](https://arxiv.org/list/cs.AI/recent) still have 50-100 papers a day. Determining if these papers are relevant and important to you means reading through the title and abstract, which is time-consuming.
+- **Multi-Model Integration**: Support for OpenAI, Gemini, and Claude models for paper analysis
+- **Latest Models**: Support for GPT-4o, GPT-4o mini, Claude 3.5, and other current models
+- **Two-Stage Processing**: Efficient paper analysis with quick filtering followed by detailed analysis
+- **Enhanced Analysis**: Detailed paper breakdowns including key innovations, critical analysis, and practical applications
+- **HTML Report Generation**: Clean, organized reports saved with date-based filenames
+- **Adjustable Relevancy Threshold**: Interactive slider for filtering papers by relevance score
+- **Design Automation Backend**: Specialized tools for analyzing design-related papers
+- **Topic Clustering**: Group similar papers using AI-powered clustering (Gemini)
+- **Robust JSON Parsing**: Reliable extraction of analysis results from LLM responses
+- **Standardized Directory Structure**: Organized codebase with `/src`, `/data`, and `/digest` directories
+- **Improved Web UI**: Clean Gradio interface with dynamic topic selection and error handling
-This repository offers a method to curate a daily digest, sorted by relevance, using large language models. These models are conditioned based on your personal research interests, which are described in natural language.
+
-* You modify the configuration file `config.yaml` with an arXiv Subject, some set of Categories, and a natural language statement about the type of papers you are interested in.
-* The code pulls all the abstracts for papers in those categories and ranks how relevant they are to your interest on a scale of 1-10 using `gpt-3.5-turbo-16k`.
-* The code then emits an HTML digest listing all the relevant papers, and optionally emails it to you using [SendGrid](https://sendgrid.com). You will need to have a SendGrid account with an API key for this functionality to work.
+## π Quick Start
-### Testing it out with Hugging Face:
+Try it out on [Hugging Face](https://huggingface.co/spaces/linhkid91/ArxivDigest-extra) using your own API keys.
-We provide a demo at [https://huggingface.co/spaces/AutoLLM/ArxivDigest](https://huggingface.co/spaces/AutoLLM/ArxivDigest). Simply enter your [OpenAI API key](https://platform.openai.com/account/api-keys) and then fill in the configuration on the right. Note that we do not store your key.
+## π What This Repo Does
-
+Staying up to date on [arXiv](https://arxiv.org) papers is time-consuming, with hundreds of new papers published daily. Even with the [official daily digest service](https://info.arxiv.org/help/subscribe.html), categories like [cs.AI](https://arxiv.org/list/cs.AI/recent) still contain 50-100 papers per day.
-You can also send yourself an email of the digest by creating a SendGrid account and [API key](https://app.SendGrid.com/settings/api_keys).
+This repository creates a personalized daily digest by:
-### Some examples of results:
+1. **Crawling arXiv** for recent papers in your areas of interest
+2. **Analyzing papers** in-depth using AI models (OpenAI, Gemini, or Claude)
+3. **Two-stage processing** for efficiency:
+ - Stage 1: Quick relevancy filtering using only title and abstract
+ - Stage 2: Detailed analysis of papers that meet the relevancy threshold
+4. **Scoring relevance** on a scale of 1-10 based on your research interests
+5. **Providing detailed analysis** of each paper, including:
+ - Key innovations
+ - Critical analysis
+ - Implementation details
+ - Practical applications
+ - Related work
+6. **Generating reports** in HTML format with clean organization
-#### Digest Configuration:
-- Subject/Topic: Computer Science
-- Categories: Artificial Intelligence, Computation and Language
-- Interest:
- - Large language model pretraining and finetunings
- - Multimodal machine learning
- - Do not care about specific application, for example, information extraction, summarization, etc.
- - Not interested in paper focus on specific languages, e.g., Arabic, Chinese, etc.
+## π€ Model Integrations
-#### Result:
-
+The system supports three major AI providers:
-#### Digest Configuration:
-- Subject/Topic: Quantitative Finance
-- Interest: "making lots of money"
+- **OpenAI GPT** (gpt-3.5-turbo-16k, gpt-4, gpt-4-turbo, gpt-4o, gpt-4o-mini)
+- **Google Gemini** (gemini-1.5-flash, gemini-1.5-pro, gemini-2.0-flash)
+- **Anthropic Claude** (claude-3-haiku, claude-3-sonnet, claude-3-opus, claude-3.5-sonnet)
-#### Result:
-
+You can use any combination of these models, allowing you to compare results or choose based on your needs.
-## π‘ Usage
+## π Output Formats
-### Running as a github action using SendGrid (Recommended).
+Reports are generated in multiple formats:
-The recommended way to get started using this repository is to:
+- **HTML Reports**: Clean, organized reports saved to the `/digest` directory with date-based filenames
+- **Console Output**: Summary information displayed in the terminal
+- **JSON Data**: Raw paper data saved to the `/data` directory
-1. Fork the repository
-2. Modify `config.yaml` and merge the changes into your main branch.
-3. Set the following secrets [(under settings, Secrets and variables, repository secrets)](https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository). See [Advanced Usage](./advanced_usage.md#create-and-fetch-your-api-keys) for more details on how to create and get OpenAi and SendGrid API keys:
- - `OPENAI_API_KEY` From [OpenAI](https://platform.openai.com/account/api-keys)
- - `SENDGRID_API_KEY` From [SendGrid](https://app.SendGrid.com/settings/api_keys)
- - `FROM_EMAIL` This value must match the email you used to create the SendGrid API Key.
- - `TO_EMAIL`
-4. Manually trigger the action or wait until the scheduled action takes place.
+Every HTML report includes:
+- Paper title, authors, and link to arXiv
+- Relevancy score with explanation
+- Abstract and key innovations
+- Critical analysis and implementation details
+- Experiments, results, and discussion points
+- Related work and practical applications
-See [Advanced Usage](./advanced_usage.md) for more details, including step-by-step images, further customization, and alternate usage.
+Example HTML report:
-### Running with a user interface
+
+## π‘ Setting Up and Usage
-To locally run the same UI as the Huggign Face space:
-
-1. Install the requirements in `src/requirements.txt` as well as `gradio`.
-2. Run `python src/app.py` and go to the local URL. From there you will be able to preview the papers from today, as well as the generated digests.
-3. If you want to use a `.env` file for your secrets, you can copy `.env.template` to `.env` and then set the environment variables in `.env`.
-- Note: These file may be hidden by default in some operating systems due to the dot prefix.
-- The .env file is one of the files in .gitignore, so git does not track it and it will not be uploaded to the repository.
-- Do not edit the original `.env.template` with your keys or your email address, since `.template.env` is tracked by git and editing it might cause you to commit your secrets.
+### Configuration
-> **WARNING:** Do not edit and commit your `.env.template` with your personal keys or email address! Doing so may expose these to the world!
+Modify `config.yaml` with your preferences:
-## β
Roadmap
+```yaml
+# Main research area
+topic: "Computer Science"
+
+# Specific categories to monitor
+categories: ["Artificial Intelligence", "Computation and Language", "Machine Learning", "Information Retrieval"]
+
+# Minimum relevance score (1-10)
+threshold: 2
+
+# Your research interests in natural language
+interest: |
+ 1. AI alignment and AI safety
+ 2. Mechanistic interpretability and explainable AI
+ 3. Large language model optimization
+ 4. RAGs, Information retrieval
+ 5. AI Red teaming, deception and misalignment
+```
+
+### Running the Web Interface
+
+To run locally with the simplified UI:
+
+1. Install requirements: `pip install -r requirements.txt`
+2. Run the app: `python src/app_new.py`
+3. Open the URL displayed in your terminal
+4. Enter your API key(s) and configure your preferences
+5. Use the relevancy threshold slider to adjust paper filtering (default is 2)
+
+### Running via GitHub Action
+
+To set up automated daily digests:
-- [x] Support personalized paper recommendation using LLM.
-- [x] Send emails for daily digest.
-- [ ] Implement a ranking factor to prioritize content from specific authors.
-- [ ] Support open-source models, e.g., LLaMA, Vicuna, MPT etc.
-- [ ] Fine-tune an open-source model to better support paper ranking and stay updated with the latest research concepts..
+1. Fork this repository
+2. Update `config.yaml` with your preferences
+3. Set the following secrets in your repository settings:
+ - `OPENAI_API_KEY` (and/or `GEMINI_API_KEY` or `ANTHROPIC_API_KEY`)
+4. The GitHub Action will run on schedule or can be triggered manually
+### Running from Command Line
-## π Extending and Contributing
+For advanced users:
-You may (and are encourage to) modify the code in this repository to suit your personal needs. If you think your modifications would be in any way useful to others, please submit a pull request.
+```bash
+# Regular paper digests with simplified UI
+python src/app_new.py
+
+# Design paper finder
+./src/design/find_design_papers.sh --days 7 --analyze
+```
+
+## β οΈ API Usage Notes
+
+This tool respects arXiv's robots.txt and implements proper rate limiting. If you encounter 403 Forbidden errors:
+
+1. Wait a few hours before trying again
+2. Consider reducing the number of categories you're fetching
+3. Increase the delay between requests in the code
+
+## π Directory Structure
+
+The repository is organized as follows:
+
+- `/src` - All Python source code
+ - `app_new.py` - Simplified interface with improved threshold handling and UI
+ - `download_new_papers.py` - arXiv crawler
+ - `relevancy.py` - Paper scoring and analysis with robust JSON parsing
+ - `model_manager.py` - Multi-model integration
+ - `gemini_utils.py` - Gemini API integration
+ - `anthropic_utils.py` - Claude API integration
+ - `design/` - Design automation tools
+ - `paths.py` - Standardized path handling
+- `/data` - JSON data files (auto-created)
+- `/digest` - HTML report files (auto-created)
+
+## β
Roadmap
-These types of modifications include things like changes to the prompt, different language models, or additional ways for the digest is delivered to you.
+- [x] Support multiple AI models (OpenAI, Gemini, Claude)
+- [x] Generate comprehensive HTML reports with date-based filenames
+- [x] Specialized analysis for design automation papers
+- [x] Topic clustering via Gemini
+- [x] Standardized directory structure
+- [x] Enhanced HTML reports with detailed analysis sections
+- [x] Pre-filtering of arXiv categories for efficiency
+- [x] Adjustable relevancy threshold with UI slider
+- [x] Robust JSON parsing for reliable LLM response handling
+- [x] Simplified UI focused on core functionality
+- [x] Dynamic topic selection UI with improved error handling
+- [x] Support for newer models (GPT-4o, GPT-4o mini, Claude 3.5)
+- [x] Two-stage paper processing for efficiency (quick filtering followed by detailed analysis)
+- [x] Removed email functionality in favor of local HTML reports
+- [ ] Full PDF content analysis
+- [ ] Author-based ranking and filtering
+- [ ] Fine-tuned open-source model support: Ollama, LocalAI...
+
+## π Contributing
+
+You're encouraged to modify this code for your personal needs. If your modifications would be useful to others, please submit a pull request.
+
+Valuable contributions include:
+- Additional AI model integrations
+- New analysis capabilities
+- UI improvements
+- Prompt engineering enhancements
diff --git a/config.yaml b/config.yaml
index feabf85..ed63acf 100644
--- a/config.yaml
+++ b/config.yaml
@@ -3,13 +3,13 @@ topic: "Computer Science"
# An empty list here will include all categories in a topic
# Use the natural language names of the topics, found here: https://arxiv.org
# Including more categories will result in more calls to the large language model
-categories: ["Artificial Intelligence", "Computation and Language"]
+categories: ["Artificial Intelligence", "Computation and Language", "Machine Learning", "Information Retrieval"]
# Relevance score threshold. abstracts that receive a score less than this from the large language model
# will have their papers filtered out.
#
# Must be within 1-10
-threshold: 7
+threshold: 2
# A natural language statement that the large language model will use to judge which papers are relevant
#
@@ -21,7 +21,10 @@ threshold: 7
# This can be empty, which just return a full list of papers with no judgement or filtering,
# in whatever order arXiv responds with.
interest: |
- 1. Large language model pretraining and finetunings
- 2. Multimodal machine learning
- 3. Do not care about specific application, for example, information extraction, summarization, etc.
- 4. Not interested in paper focus on specific languages, e.g., Arabic, Chinese, etc.
+ 1. AI alignment and AI safety
+ 2. Mechanistic interpretability and explainable AI
+ 3. Large language model under pressure
+ 4. AI Red teaming, deception and misalignment
+ 5. RAGs, Information retrieval
+ 6. Optimization of LLM and GenAI
+ 7. Do not care about specific application, for example, information extraction, summarization, etc.
diff --git a/digest/design_papers.html b/digest/design_papers.html
new file mode 100644
index 0000000..90f98f6
--- /dev/null
+++ b/digest/design_papers.html
@@ -0,0 +1,177 @@
+
+
+
+
+
+ Design Automation Papers
+
+
+
+ Design Automation Papers
+
+
Found 18 papers related to graphic design automation with AI/ML
+
Generated on 2025-04-06 13:57:23
+
Summary Statistics
Categories:
- Layout Generation: 7 papers
- Layout Generation, UI/UX Design: 6 papers
- Layout Generation, UI/UX Design, Image Manipulation: 1 papers
- UI/UX Design, Design Tools: 1 papers
- Layout Generation, UI/UX Design, 3D Design: 1 papers
- Layout Generation, Multimodal Design: 1 papers
- Layout Generation, UI/UX Design, Design Tools: 1 papers
Techniques:
- Reinforcement Learning: 11 papers
- Transformers: 4 papers
- Computer Vision: 3 papers
- Large Language Models: 3 papers
- Diffusion Models: 2 papers
+
+
Jinqi Luo, Tianjiao Ding, Kwan Ho Ryan Chan, Hancheng Min, Chris Callison-Burch, RenΓ© Vidal
+
Category: Layout Generation, UI/UX Design, Image Manipulation | Subject: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
+
Techniques: Diffusion Models
+
Abstract: Diffusion models are widely used for image editing tasks. Existing editing methods often design a representation manipulation procedure by curating an edit direction in the text embedding or score space. However, such a procedure faces a key challenge: overestimating the edit strength harms visual consistency while underestimating it fails the editing task. Notably, each source image may require a different editing strength, and it is costly to search for an appropriate strength via trial-and-error. To address this challenge, we propose Concept Lancet (CoLan), a zero-shot plug-and-play framework for principled representation manipulation in diffusion-based image editing. At inference time, we decompose the source input in the latent (text embedding or diffusion score) space as a sparse linear combination of the representations of the collected visual concepts. This allows us to accurately estimate the presence of concepts in each image, which informs the edit. Based on the editing task (replace/add/remove), we perform a customized concept transplant process to impose the corresponding editing direction. To sufficiently model the concept space, we curate a conceptual representation dataset, CoLan-150K, which contains diverse descriptions and scenarios of visual terms and phrases for the latent dictionary. Experiments on multiple diffusion-based image editing baselines show that methods equipped with CoLan achieve state-of-the-art performance in editing effectiveness and consistency preservation.
+
+
+
Yuhang Li, Ruokai Yin, Donghyun Lee, Shiting Xiao, Priyadarshini Panda
+
Category: Layout Generation | Subject: Machine Learning (cs.LG)
+
Techniques: Transformers, Reinforcement Learning, Computer Vision, Large Language Models
+
Abstract: We introduce GPTQv2, a novel finetuning-free quantization method for compressing large-scale transformer architectures. Unlike the previous GPTQ method, which independently calibrates each layer, we always match the quantized layer's output to the exact output in the full-precision model, resulting in a scheme that we call asymmetric calibration. Such a scheme can effectively reduce the quantization error accumulated in previous layers. We analyze this problem using optimal brain compression to derive a close-formed solution. The new solution explicitly minimizes the quantization error as well as the accumulated asymmetry error. Furthermore, we utilize various techniques to parallelize the solution calculation, including channel parallelization, neuron decomposition, and Cholesky reformulation for matrix fusion. As a result, GPTQv2 is easy to implement, simply using 20 more lines of code than GPTQ but improving its performance under low-bit quantization. Remarkably, on a single GPU, we quantize a 405B language transformer as well as EVA-02 the rank first vision transformer that achieves 90% pretraining Imagenet accuracy. Code is available at this http URL.
+
+
+
Thomas Dooms, Ward Gauderis, Geraint A. Wiggins, Jose Oramas
+
Category: Layout Generation | Subject: Machine Learning (cs.LG)
+
Techniques:
+
Abstract: We propose $\chi$-net, an intrinsically interpretable architecture combining the compositional multilinear structure of tensor networks with the expressivity and efficiency of deep neural networks. $\chi$-nets retain equal accuracy compared to their baseline counterparts. Our novel, efficient diagonalisation algorithm, ODT, reveals linear low-rank structure in a multilayer SVHN model. We leverage this toward formal weight-based interpretability and model compression.
+
+
+
Leonardo Iurada, Marco Ciccone, Tatiana Tommasi
+
Category: Layout Generation, UI/UX Design | Subject: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV)
+
Techniques: Reinforcement Learning
+
Abstract: Task arithmetic has emerged as a promising approach for editing models by representing task-specific knowledge as composable task vectors. However, existing methods rely on network linearization to derive task vectors, leading to computational bottlenecks during training and inference. Moreover, linearization alone does not ensure weight disentanglement, the key property that enables conflict-free composition of task vectors. To address this, we propose TaLoS which allows to build sparse task vectors with minimal interference without requiring explicit linearization and sharing information across tasks. We find that pre-trained models contain a subset of parameters with consistently low gradient sensitivity across tasks, and that sparsely updating only these parameters allows for promoting weight disentanglement during fine-tuning. Our experiments prove that TaLoS improves training and inference efficiency while outperforming current methods in task addition and negation. By enabling modular parameter editing, our approach fosters practical deployment of adaptable foundation models in real-world applications.
+
+
+
Viacheslav Yusupov, Maxim Rakhuba, Evgeny Frolov
+
Category: Layout Generation | Subject: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Information Retrieval (cs.IR); Machine Learning (stat.ML)
+
Techniques: Reinforcement Learning
+
Abstract: In this paper, we propose a new geometric approach for knowledge graph completion via low rank tensor approximation. We augment a pretrained and well-established Euclidean model based on a Tucker tensor decomposition with a novel hyperbolic interaction term. This correction enables more nuanced capturing of distributional properties in data better aligned with real-world knowledge graphs. By combining two geometries together, our approach improves expressivity of the resulting model achieving new state-of-the-art link prediction accuracy with a significantly lower number of parameters compared to the previous Euclidean and hyperbolic models.
+
+
+
Siddharth Khincha, Tushar Kataria, Ankita Anand, Dan Roth, Vivek Gupta
+
Category: Layout Generation | Subject: Computation and Language (cs.CL)
+
Techniques: Reinforcement Learning, Large Language Models
+
Abstract: The vast amount of online information today poses challenges for non-English speakers, as much of it is concentrated in high-resource languages such as English and French. Wikipedia reflects this imbalance, with content in low-resource languages frequently outdated or incomplete. Recent research has sought to improve cross-language synchronization of Wikipedia tables using rule-based methods. These approaches can be effective, but they struggle with complexity and generalization. This paper explores large language models (LLMs) for multilingual information synchronization, using zero-shot prompting as a scalable solution. We introduce the Information Updation dataset, simulating the real-world process of updating outdated Wikipedia tables, and evaluate LLM performance. Our findings reveal that single-prompt approaches often produce suboptimal results, prompting us to introduce a task decomposition strategy that enhances coherence and accuracy. Our proposed method outperforms existing baselines, particularly in Information Updation (1.79%) and Information Addition (20.58%), highlighting the model strength in dynamically updating and enriching data across architectures
+
+
+
Jeba Rezwana, Corey Ford
+
Category: UI/UX Design, Design Tools | Subject: Human-Computer Interaction (cs.HC); Artificial Intelligence (cs.AI)
+
Techniques:
+
Abstract: How AI communicates with humans is crucial for effective human-AI co-creation. However, many existing co-creative AI tools cannot communicate effectively, limiting their potential as collaborators. This paper introduces our initial design of a Framework for designing AI Communication (FAICO) for co-creative AI based on a systematic review of 107 full-length papers. FAICO presents key aspects of AI communication and their impacts on user experience to guide the design of effective AI communication. We then show actionable ways to translate our framework into two practical tools: design cards for designers and a configuration tool for users. The design cards enable designers to consider AI communication strategies that cater to a diverse range of users in co-creative contexts, while the configuration tool empowers users to customize AI communication based on their needs and creative workflows. This paper contributes new insights within the literature on human-AI co-creativity and Human-Computer Interaction, focusing on designing AI communication to enhance user experience.
+
+
+
Fatemeh Behrad, Tinne Tuytelaars, Johan Wagemans
+
Category: Layout Generation | Subject: Computer Vision and Pattern Recognition (cs.CV)
+
Techniques: Transformers, Reinforcement Learning, Computer Vision
+
Abstract: The capacity of Vision transformers (ViTs) to handle variable-sized inputs is often constrained by computational complexity and batch processing limitations. Consequently, ViTs are typically trained on small, fixed-size images obtained through downscaling or cropping. While reducing computational burden, these methods result in significant information loss, negatively affecting tasks like image aesthetic assessment. We introduce Charm, a novel tokenization approach that preserves Composition, High-resolution, Aspect Ratio, and Multi-scale information simultaneously. Charm prioritizes high-resolution details in specific regions while downscaling others, enabling shorter fixed-size input sequences for ViTs while incorporating essential information. Charm is designed to be compatible with pre-trained ViTs and their learned positional embeddings. By providing multiscale input and introducing variety to input tokens, Charm improves ViT performance and generalizability for image aesthetic assessment. We avoid cropping or changing the aspect ratio to further preserve information. Extensive experiments demonstrate significant performance improvements on various image aesthetic and quality assessment datasets (up to 8.1 %) using a lightweight ViT backbone. Code and pre-trained models are available at this https URL.
+
+
+
Sinchee Chin, Fan Zhang, Xiaochen Yang, Jing-Hao Xue, Wenming Yang, Peng Jia, Guijin Wang, Luo Yingqun
+
Category: Layout Generation, UI/UX Design, 3D Design | Subject: Machine Learning (cs.LG); Information Theory (cs.IT)
+
Techniques: Reinforcement Learning
+
Abstract: Time Series Anomaly Detection (TSAD) is essential for uncovering rare and potentially harmful events in unlabeled time series data. Existing methods are highly dependent on clean, high-quality inputs, making them susceptible to noise and real-world imperfections. Additionally, intricate temporal relationships in time series data are often inadequately captured in traditional 1D representations, leading to suboptimal modeling of dependencies. We introduce VISTA, a training-free, unsupervised TSAD algorithm designed to overcome these challenges. VISTA features three core modules: 1) Time Series Decomposition using Seasonal and Trend Decomposition via Loess (STL) to decompose noisy time series into trend, seasonal, and residual components; 2) Temporal Self-Attention, which transforms 1D time series into 2D temporal correlation matrices for richer dependency modeling and anomaly detection; and 3) Multivariate Temporal Aggregation, which uses a pretrained feature extractor to integrate cross-variable information into a unified, memory-efficient representation. VISTA's training-free approach enables rapid deployment and easy hyperparameter tuning, making it suitable for industrial applications. It achieves state-of-the-art performance on five multivariate TSAD benchmarks.
+
+
+
Qisheng Hu, Quanyu Long, Wenya Wang
+
Category: Layout Generation, UI/UX Design | Subject: Artificial Intelligence (cs.AI)
+
Techniques: Reinforcement Learning
+
Abstract: Program-guided reasoning has shown promise in complex claim fact-checking by decomposing claims into function calls and executing reasoning programs. However, prior work primarily relies on few-shot in-context learning (ICL) with ad-hoc demonstrations, which limit program diversity and require manual design with substantial domain knowledge. Fundamentally, the underlying principles of effective reasoning program generation still remain underexplored, making it challenging to construct effective demonstrations. To address this, we propose BOOST, a bootstrapping-based framework for few-shot reasoning program generation. BOOST explicitly integrates claim decomposition and information-gathering strategies as structural guidance for program generation, iteratively refining bootstrapped demonstrations in a strategy-driven and data-centric manner without human intervention. This enables a seamless transition from zero-shot to few-shot strategic program-guided learning, enhancing interpretability and effectiveness. Experimental results show that BOOST outperforms prior few-shot baselines in both zero-shot and few-shot settings for complex claim verification.
+
+
+
Jiayi Gao, Zijin Yin, Changcheng Hua, Yuxin Peng, Kongming Liang, Zhanyu Ma, Jun Guo, Yang Liu
+
Category: Layout Generation, UI/UX Design | Subject: Computer Vision and Pattern Recognition (cs.CV)
+
Techniques: Reinforcement Learning
+
Abstract: The development of Text-to-Video (T2V) generation has made motion transfer possible, enabling the control of video motion based on existing footage. However, current methods have two limitations: 1) struggle to handle multi-subjects videos, failing to transfer specific subject motion; 2) struggle to preserve the diversity and accuracy of motion as transferring to subjects with varying shapes. To overcome these, we introduce \textbf{ConMo}, a zero-shot framework that disentangle and recompose the motions of subjects and camera movements. ConMo isolates individual subject and background motion cues from complex trajectories in source videos using only subject masks, and reassembles them for target video generation. This approach enables more accurate motion control across diverse subjects and improves performance in multi-subject scenarios. Additionally, we propose soft guidance in the recomposition stage which controls the retention of original motion to adjust shape constraints, aiding subject shape adaptation and semantic transformation. Unlike previous methods, ConMo unlocks a wide range of applications, including subject size and position editing, subject removal, semantic modifications, and camera motion simulation. Extensive experiments demonstrate that ConMo significantly outperforms state-of-the-art methods in motion fidelity and semantic consistency. The code is available at this https URL.
+
+
+
Zhengcong Fei, Debang Li, Di Qiu, Jiahua Wang, Yikun Dou, Rui Wang, Jingtao Xu, Mingyuan Fan, Guibin Chen, Yang Li, Yahui Zhou
+
Category: Layout Generation | Subject: Computer Vision and Pattern Recognition (cs.CV)
+
Techniques: Diffusion Models, Transformers
+
Abstract: This paper presents SkyReels-A2, a controllable video generation framework capable of assembling arbitrary visual elements (e.g., characters, objects, backgrounds) into synthesized videos based on textual prompts while maintaining strict consistency with reference images for each element. We term this task elements-to-video (E2V), whose primary challenges lie in preserving the fidelity of each reference element, ensuring coherent composition of the scene, and achieving natural outputs. To address these, we first design a comprehensive data pipeline to construct prompt-reference-video triplets for model training. Next, we propose a novel image-text joint embedding model to inject multi-element representations into the generative process, balancing element-specific consistency with global coherence and text alignment. We also optimize the inference pipeline for both speed and output stability. Moreover, we introduce a carefully curated benchmark for systematic evaluation, i.e, A2 Bench. Experiments demonstrate that our framework can generate diverse, high-quality videos with precise element control. SkyReels-A2 is the first open-source commercial grade model for the generation of E2V, performing favorably against advanced closed-source commercial models. We anticipate SkyReels-A2 will advance creative applications such as drama and virtual e-commerce, pushing the boundaries of controllable video generation.
+
+
+
Aidan Tiruvan
+
Category: Layout Generation | Subject: Machine Learning (cs.LG); Numerical Analysis (math.NA)
+
Techniques: Reinforcement Learning
+
Abstract: Robust low-rank approximation under row-wise adversarial corruption can be achieved with a single pass, randomized procedure that detects and removes outlier rows by thresholding their projected norms. We propose a scalable, non-iterative algorithm that efficiently recovers the underlying low-rank structure in the presence of row-wise adversarial corruption. By first compressing the data with a Johnson Lindenstrauss projection, our approach preserves the geometry of clean rows while dramatically reducing dimensionality. Robust statistical techniques based on the median and median absolute deviation then enable precise identification and removal of outlier rows with abnormally high norms. The subsequent rank-k approximation achieves near-optimal error bounds with a one pass procedure that scales linearly with the number of observations. Empirical results confirm that combining random sketches with robust statistics yields efficient, accurate decompositions even in the presence of large fractions of corrupted rows.
+
+
+
Takahiro Shirakawa, Tomoyuki Suzuki, Daichi Haraguchi
+
Category: Layout Generation, UI/UX Design | Subject: Graphics (cs.GR); Computer Vision and Pattern Recognition (cs.CV)
+
Techniques:
+
Abstract: General image-to-video generation methods often produce suboptimal animations that do not meet the requirements of animated graphics, as they lack active text motion and exhibit object distortion. Also, code-based animation generation methods typically require layer-structured vector data which are often not readily available for motion graphic generation. To address these challenges, we propose a novel framework named MG-Gen that reconstructs data in vector format from a single raster image to extend the capabilities of code-based methods to enable motion graphics generation from a raster image in the framework of general image-to-video generation. MG-Gen first decomposes the input image into layer-wise elements, reconstructs them as HTML format data and then generates executable JavaScript code for the reconstructed HTML data. We experimentally confirm that \ours{} generates motion graphics while preserving text readability and input consistency. These successful results indicate that combining layer decomposition and animation code generation is an effective strategy for motion graphics generation.
+
+
+
Weibin Liao, Xin Gao, Tianyu Jia, Rihong Qiu, Yifan Zhu, Yang Lin, Xu Chu, Junfeng Zhao, Yasha Wang
+
Category: Layout Generation, UI/UX Design | Subject: Computation and Language (cs.CL)
+
Techniques: Reinforcement Learning, Large Language Models
+
Abstract: Natural Language to SQL (NL2SQL) has emerged as a critical task for enabling seamless interaction with databases. Recent advancements in Large Language Models (LLMs) have demonstrated remarkable performance in this domain. However, existing NL2SQL methods predominantly rely on closed-source LLMs leveraging prompt engineering, while open-source models typically require fine-tuning to acquire domain-specific knowledge. Despite these efforts, open-source LLMs struggle with complex NL2SQL tasks due to the indirect expression of user query objectives and the semantic gap between user queries and database schemas. Inspired by the application of reinforcement learning in mathematical problem-solving to encourage step-by-step reasoning in LLMs, we propose LearNAT (Learning NL2SQL with AST-guided Task Decomposition), a novel framework that improves the performance of open-source LLMs on complex NL2SQL tasks through task decomposition and reinforcement learning. LearNAT introduces three key components: (1) a Decomposition Synthesis Procedure that leverages Abstract Syntax Trees (ASTs) to guide efficient search and pruning strategies for task decomposition, (2) Margin-aware Reinforcement Learning, which employs fine-grained step-level optimization via DPO with AST margins, and (3) Adaptive Demonstration Reasoning, a mechanism for dynamically selecting relevant examples to enhance decomposition capabilities. Extensive experiments on two benchmark datasets, Spider and BIRD, demonstrate that LearNAT enables a 7B-parameter open-source LLM to achieve performance comparable to GPT-4, while offering improved efficiency and accessibility.
+
+
+
Zhipu Cui, Andong Tian, Zhi Ying, Jialiang Lu
+
Category: Layout Generation, Multimodal Design | Subject: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
+
Techniques:
+
Abstract: Personalized image generation allows users to preserve styles or subjects of a provided small set of images for further image generation. With the advancement in large text-to-image models, many techniques have been developed to efficiently fine-tune those models for personalization, such as Low Rank Adaptation (LoRA). However, LoRA-based methods often face the challenge of adjusting the rank parameter to achieve satisfactory results. To address this challenge, AutoComponent-LoRA (AC-LoRA) is proposed, which is able to automatically separate the signal component and noise component of the LoRA matrices for fast and efficient personalized artistic style image generation. This method is based on Singular Value Decomposition (SVD) and dynamic heuristics to update the hyperparameters during training. Superior performance over existing methods in overcoming model underfitting or overfitting problems is demonstrated. The results were validated using FID, CLIP, DINO, and ImageReward, achieving an average of 9% improvement.
+
+
+
Reza Sameni
+
Category: Layout Generation, UI/UX Design, Design Tools | Subject: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Statistics Theory (math.ST); Machine Learning (stat.ML)
+
Techniques: Reinforcement Learning
+
Abstract: We study the geometry of Receiver Operating Characteristic (ROC) and Precision-Recall (PR) curves in binary classification problems. The key finding is that many of the most commonly used binary classification metrics are merely functions of the composition function $G := F_p \circ F_n^{-1}$, where $F_p(\cdot)$ and $F_n(\cdot)$ are the class-conditional cumulative distribution functions of the classifier scores in the positive and negative classes, respectively. This geometric perspective facilitates the selection of operating points, understanding the effect of decision thresholds, and comparison between classifiers. It also helps explain how the shapes and geometry of ROC/PR curves reflect classifier behavior, providing objective tools for building classifiers optimized for specific applications with context-specific constraints. We further explore the conditions for classifier dominance, present analytical and numerical examples demonstrating the effects of class separability and variance on ROC and PR geometries, and derive a link between the positive-to-negative class leakage function $G(\cdot)$ and the Kullback--Leibler divergence. The framework highlights practical considerations, such as model calibration, cost-sensitive optimization, and operating point selection under real-world capacity constraints, enabling more informed approaches to classifier deployment and decision-making.
+
+
+
Zechen Liu, Feiyang Zhang, Wei Song, Xiang Li, Wei Wei
+
Category: Layout Generation, UI/UX Design | Subject: Machine Learning (cs.LG)
+
Techniques: Transformers, Computer Vision
+
Abstract: The study of neural networks from the perspective of Fourier features has garnered significant attention. While existing analytical research suggests that neural networks tend to learn low-frequency features, a clear attribution method for identifying the specific learned Fourier features has remained elusive. To bridge this gap, we propose a novel Fourier feature attribution method grounded in signal decomposition theory. Additionally, we analyze the differences between game-theoretic attribution metrics for Fourier and spatial domain features, demonstrating that game-theoretic evaluation metrics are better suited for Fourier-based feature attribution. Our experiments show that Fourier feature attribution exhibits superior feature selection capabilities compared to spatial domain attribution methods. For instance, in the case of Vision Transformers (ViTs) on the ImageNet dataset, only $8\%$ of the Fourier features are required to maintain the original predictions for $80\%$ of the samples. Furthermore, we compare the specificity of features identified by our method against traditional spatial domain attribution methods. Results reveal that Fourier features exhibit greater intra-class concentration and inter-class distinctiveness, indicating their potential for more efficient classification and explainable AI algorithms.
+
".join(
[
- f'Title: {paper["title"]}
Authors: {paper["authors"]}
Score: {paper["Relevancy score"]}
Reason: {paper["Reasons for match"]}'
+ f'Subject: {paper["subjects"]}
Title: {paper["title"]}
Authors: {paper["authors"]}
'
+ f'Score: {paper["Relevancy score"]}
Reason: {paper["Reasons for match"]}
'
+ f'Goal: {paper["Goal"]}
Data: {paper["Data"]}
Methodology: {paper["Methodology"]}
'
+ f'Experiments & Results: {paper["Experiments & Results"]}
Git: {paper["Git"]}
'
+ f'Discussion & Next steps: {paper["Discussion & Next steps"]}'
for paper in relevancy
]
)
@@ -269,6 +279,10 @@ def generate_body(topic, categories, interest, threshold):
)
return body
+def get_date():
+ today = date.today()
+ formatted_date = today.strftime("%d%m%Y")
+ return formatted_date
if __name__ == "__main__":
# Load the .env file.
@@ -292,7 +306,8 @@ def generate_body(topic, categories, interest, threshold):
threshold = config["threshold"]
interest = config["interest"]
body = generate_body(topic, categories, interest, threshold)
- with open("digest.html", "w") as f:
+ today_date = get_date()
+ with open(f"digest_{today_date}.html", "w") as f:
f.write(body)
if os.environ.get("SENDGRID_API_KEY", None):
sg = SendGridAPIClient(api_key=os.environ.get("SENDGRID_API_KEY"))
diff --git a/src/anthropic_utils.py b/src/anthropic_utils.py
new file mode 100644
index 0000000..e6ef0e7
--- /dev/null
+++ b/src/anthropic_utils.py
@@ -0,0 +1,322 @@
+"""
+Anthropic/Claude API integration for ArxivDigest.
+This module provides functions to work with Anthropic's Claude API for paper analysis.
+"""
+import os
+import json
+import logging
+import time
+from typing import List, Dict, Any, Optional
+
+try:
+ import anthropic
+ from anthropic.types import MessageParam
+ ANTHROPIC_AVAILABLE = True
+except ImportError:
+ ANTHROPIC_AVAILABLE = False
+
+# Configure logging
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger(__name__)
+
+class ClaudeConfig:
+ """Configuration for Claude API calls."""
+ def __init__(
+ self,
+ temperature: float = 0.5,
+ max_tokens: int = 4000,
+ top_p: float = 0.95,
+ top_k: int = 40
+ ):
+ self.temperature = temperature
+ self.max_tokens = max_tokens
+ self.top_p = top_p
+ self.top_k = top_k
+
+def setup_anthropic_api(api_key: str) -> bool:
+ """
+ Setup the Anthropic API with the provided API key.
+
+ Args:
+ api_key: Anthropic API key
+
+ Returns:
+ bool: True if setup was successful, False otherwise
+ """
+ if not ANTHROPIC_AVAILABLE:
+ logger.error("Anthropic package not installed. Run 'pip install anthropic'")
+ return False
+
+ if not api_key:
+ logger.error("No Anthropic API key provided")
+ return False
+
+ try:
+ # Initialize client to test connection
+ client = anthropic.Anthropic(api_key=api_key)
+ # Test API connection by listing models
+ models = client.models.list()
+ available_models = [model.id for model in models.data]
+ logger.info(f"Successfully connected to Anthropic API. Available models: {available_models}")
+ return True
+ except Exception as e:
+ logger.error(f"Failed to setup Anthropic API: {e}")
+ return False
+
+def get_claude_client(api_key: str) -> Optional[anthropic.Anthropic]:
+ """
+ Get an Anthropic client with the given API key.
+
+ Args:
+ api_key: Anthropic API key
+
+ Returns:
+ Anthropic client or None if not available
+ """
+ if not ANTHROPIC_AVAILABLE:
+ return None
+
+ try:
+ client = anthropic.Anthropic(api_key=api_key)
+ return client
+ except Exception as e:
+ logger.error(f"Failed to get Anthropic client: {e}")
+ return None
+
+def analyze_papers_with_claude(
+ papers: List[Dict[str, Any]],
+ query: Dict[str, str],
+ config: Optional[ClaudeConfig] = None,
+ model_name: str = "claude-3.5-sonnet-20240620",
+ api_key: str = None
+) -> List[Dict[str, Any]]:
+ """
+ Analyze papers using Claude.
+
+ Args:
+ papers: List of paper dictionaries
+ query: Dictionary with 'interest' key describing research interests
+ config: ClaudeConfig object
+ model_name: Name of the Claude model to use
+ api_key: Anthropic API key (optional if already configured elsewhere)
+
+ Returns:
+ List of papers with added analysis
+ """
+ if not ANTHROPIC_AVAILABLE:
+ logger.error("Anthropic package not installed. Cannot analyze papers.")
+ return papers
+
+ if not config:
+ config = ClaudeConfig()
+
+ # Get client
+ if api_key:
+ client = get_claude_client(api_key)
+ else:
+ # Try to get from environment
+ api_key = os.environ.get("ANTHROPIC_API_KEY", "")
+ if not api_key:
+ logger.error("No Anthropic API key provided")
+ return papers
+ client = get_claude_client(api_key)
+
+ if not client:
+ return papers
+
+ analyzed_papers = []
+
+ for paper in papers:
+ try:
+ # Prepare system prompt
+ system_prompt = f"""
+ You are a research assistant analyzing academic papers in AI and ML.
+ You provide comprehensive, accurate and unbiased analysis based on the user's research interests.
+ Your responses should be well-structured and factual, focusing on the paper's strengths, weaknesses, and relevance.
+ """
+
+ # Prepare user prompt
+ user_prompt = f"""
+ Analyze this paper and provide insights based on the following research interests:
+
+ Research interests: {query['interest']}
+
+ Paper details:
+ Title: {paper['title']}
+ Authors: {paper['authors']}
+ Abstract: {paper['abstract']}
+ Content: {paper['content'][:5000] if 'content' in paper else 'Not available'}
+
+ Please provide your response as a single JSON object with the following structure:
+ {{
+ "Relevancy score": 1-10 (higher = more relevant),
+ "Reasons for match": "Detailed explanation of why this paper matches the interests",
+ "Key innovations": "List the main contributions of the paper",
+ "Critical analysis": "Evaluate strengths and weaknesses",
+ "Goal": "What problem does the paper address?",
+ "Data": "Description of datasets used",
+ "Methodology": "Technical approach and methods",
+ "Implementation details": "Model architecture, hyperparameters, etc.",
+ "Experiments & Results": "Key findings and comparisons",
+ "Discussion & Next steps": "Limitations and future work",
+ "Related work": "Connection to similar research",
+ "Practical applications": "Real-world uses of this research",
+ "Key takeaways": ["Point 1", "Point 2", "Point 3"]
+ }}
+
+ Format your response as a valid JSON object and nothing else.
+ """
+
+ # Just log that we're sending a prompt to Claude
+ print(f"Sending prompt to Claude for paper: {paper['title'][:50]}...")
+
+ # Create message
+ messages: List[MessageParam] = [
+ {
+ "role": "user",
+ "content": user_prompt
+ }
+ ]
+
+ # Call the API
+ response = client.messages.create(
+ model=model_name,
+ max_tokens=config.max_tokens,
+ temperature=config.temperature,
+ system=system_prompt,
+ messages=messages
+ )
+
+ # Extract and parse the response
+ response_text = response.content[0].text if response.content else ""
+
+ # Try to extract JSON
+ try:
+ start_idx = response_text.find('{')
+ end_idx = response_text.rfind('}') + 1
+ if start_idx >= 0 and end_idx > start_idx:
+ json_str = response_text[start_idx:end_idx]
+ claude_analysis = json.loads(json_str)
+
+ # Add Claude analysis to paper
+ paper['claude_analysis'] = claude_analysis
+
+ # Directly copy fields to paper
+ for key, value in claude_analysis.items():
+ paper[key] = value
+ else:
+ logger.warning(f"Could not extract JSON from Claude response for paper {paper['title']}")
+ paper['claude_analysis'] = {"error": "Failed to parse response"}
+ except json.JSONDecodeError:
+ logger.warning(f"Failed to parse Claude response as JSON for paper {paper['title']}")
+ paper['claude_analysis'] = {"error": "Failed to parse response"}
+
+ analyzed_papers.append(paper)
+
+ # Avoid rate limiting
+ time.sleep(1)
+
+ except Exception as e:
+ logger.error(f"Claude API error: {e}")
+ paper['claude_analysis'] = {"error": f"Claude API error: {str(e)}"}
+ analyzed_papers.append(paper)
+
+ return analyzed_papers
+
+def get_claude_interpretability_analysis(paper: Dict[str, Any], model_name: str = "claude-3.5-sonnet-20240620", api_key: str = None) -> Dict[str, Any]:
+ """
+ Get specialized mechanistic interpretability analysis for a paper using Claude.
+
+ Args:
+ paper: Paper dictionary
+ model_name: Claude model to use
+ api_key: Anthropic API key (optional if already configured elsewhere)
+
+ Returns:
+ Dictionary with interpretability analysis
+ """
+ if not ANTHROPIC_AVAILABLE:
+ return {"error": "Anthropic package not installed"}
+
+ # Get client
+ if api_key:
+ client = get_claude_client(api_key)
+ else:
+ # Try to get from environment
+ api_key = os.environ.get("ANTHROPIC_API_KEY", "")
+ if not api_key:
+ return {"error": "No Anthropic API key provided"}
+ client = get_claude_client(api_key)
+
+ if not client:
+ return {"error": "Failed to initialize Anthropic client"}
+
+ try:
+ # Prepare system prompt
+ system_prompt = """
+ You are a specialist in mechanistic interpretability and AI alignment.
+ Provide a thorough analysis of research papers with focus on interpretability methods,
+ circuit analysis, and how the work relates to understanding AI systems.
+ """
+
+ # Prepare the prompt
+ user_prompt = f"""
+ Analyze this paper from a mechanistic interpretability perspective:
+
+ Title: {paper['title']}
+ Authors: {paper['authors']}
+ Abstract: {paper['abstract']}
+ Content: {paper['content'][:7000] if 'content' in paper else paper['abstract']}
+
+ Please return your analysis as a JSON object with the following fields:
+
+ {{
+ "interpretability_score": 1-10 (how relevant is this to mechanistic interpretability),
+ "key_methods": "Main interpretability techniques used or proposed",
+ "circuit_analysis": "Any findings about neural circuits or components",
+ "relevance_to_alignment": "How this work contributes to AI alignment",
+ "novel_insights": "New perspectives on model internals",
+ "limitations": "Limitations of the interpretability methods",
+ "potential_extensions": "How this work could be extended",
+ "connection_to_other_work": "Relationship to other interpretability papers"
+ }}
+
+ Respond with only the JSON.
+ """
+
+ # Create message
+ messages: List[MessageParam] = [
+ {
+ "role": "user",
+ "content": user_prompt
+ }
+ ]
+
+ # Call the API
+ response = client.messages.create(
+ model=model_name,
+ max_tokens=4000,
+ temperature=0.3,
+ system=system_prompt,
+ messages=messages
+ )
+
+ # Extract and parse the response
+ response_text = response.content[0].text if response.content else ""
+
+ # Try to extract JSON
+ try:
+ # Find the JSON part in the response
+ start_idx = response_text.find('{')
+ end_idx = response_text.rfind('}') + 1
+ if start_idx >= 0 and end_idx > start_idx:
+ json_str = response_text[start_idx:end_idx]
+ analysis = json.loads(json_str)
+ return analysis
+ else:
+ return {"error": "Could not extract JSON from response"}
+ except json.JSONDecodeError:
+ return {"error": "Failed to parse response as JSON"}
+
+ except Exception as e:
+ return {"error": f"Claude API error: {str(e)}"}
\ No newline at end of file
diff --git a/src/app.py b/src/app.py
deleted file mode 100644
index 0743ad5..0000000
--- a/src/app.py
+++ /dev/null
@@ -1,195 +0,0 @@
-import gradio as gr
-from download_new_papers import get_papers
-import utils
-from relevancy import generate_relevance_score, process_subject_fields
-from sendgrid.helpers.mail import Mail, Email, To, Content
-import sendgrid
-import os
-import openai
-
-topics = {
- "Physics": "",
- "Mathematics": "math",
- "Computer Science": "cs",
- "Quantitative Biology": "q-bio",
- "Quantitative Finance": "q-fin",
- "Statistics": "stat",
- "Electrical Engineering and Systems Science": "eess",
- "Economics": "econ"
-}
-
-physics_topics = {
- "Astrophysics": "astro-ph",
- "Condensed Matter": "cond-mat",
- "General Relativity and Quantum Cosmology": "gr-qc",
- "High Energy Physics - Experiment": "hep-ex",
- "High Energy Physics - Lattice": "hep-lat",
- "High Energy Physics - Phenomenology": "hep-ph",
- "High Energy Physics - Theory": "hep-th",
- "Mathematical Physics": "math-ph",
- "Nonlinear Sciences": "nlin",
- "Nuclear Experiment": "nucl-ex",
- "Nuclear Theory": "nucl-th",
- "Physics": "physics",
- "Quantum Physics": "quant-ph"
-}
-
-categories_map = {
- "Astrophysics": ["Astrophysics of Galaxies", "Cosmology and Nongalactic Astrophysics", "Earth and Planetary Astrophysics", "High Energy Astrophysical Phenomena", "Instrumentation and Methods for Astrophysics", "Solar and Stellar Astrophysics"],
- "Condensed Matter": ["Disordered Systems and Neural Networks", "Materials Science", "Mesoscale and Nanoscale Physics", "Other Condensed Matter", "Quantum Gases", "Soft Condensed Matter", "Statistical Mechanics", "Strongly Correlated Electrons", "Superconductivity"],
- "General Relativity and Quantum Cosmology": ["None"],
- "High Energy Physics - Experiment": ["None"],
- "High Energy Physics - Lattice": ["None"],
- "High Energy Physics - Phenomenology": ["None"],
- "High Energy Physics - Theory": ["None"],
- "Mathematical Physics": ["None"],
- "Nonlinear Sciences": ["Adaptation and Self-Organizing Systems", "Cellular Automata and Lattice Gases", "Chaotic Dynamics", "Exactly Solvable and Integrable Systems", "Pattern Formation and Solitons"],
- "Nuclear Experiment": ["None"],
- "Nuclear Theory": ["None"],
- "Physics": ["Accelerator Physics", "Applied Physics", "Atmospheric and Oceanic Physics", "Atomic and Molecular Clusters", "Atomic Physics", "Biological Physics", "Chemical Physics", "Classical Physics", "Computational Physics", "Data Analysis, Statistics and Probability", "Fluid Dynamics", "General Physics", "Geophysics", "History and Philosophy of Physics", "Instrumentation and Detectors", "Medical Physics", "Optics", "Physics and Society", "Physics Education", "Plasma Physics", "Popular Physics", "Space Physics"],
- "Quantum Physics": ["None"],
- "Mathematics": ["Algebraic Geometry", "Algebraic Topology", "Analysis of PDEs", "Category Theory", "Classical Analysis and ODEs", "Combinatorics", "Commutative Algebra", "Complex Variables", "Differential Geometry", "Dynamical Systems", "Functional Analysis", "General Mathematics", "General Topology", "Geometric Topology", "Group Theory", "History and Overview", "Information Theory", "K-Theory and Homology", "Logic", "Mathematical Physics", "Metric Geometry", "Number Theory", "Numerical Analysis", "Operator Algebras", "Optimization and Control", "Probability", "Quantum Algebra", "Representation Theory", "Rings and Algebras", "Spectral Theory", "Statistics Theory", "Symplectic Geometry"],
- "Computer Science": ["Artificial Intelligence", "Computation and Language", "Computational Complexity", "Computational Engineering, Finance, and Science", "Computational Geometry", "Computer Science and Game Theory", "Computer Vision and Pattern Recognition", "Computers and Society", "Cryptography and Security", "Data Structures and Algorithms", "Databases", "Digital Libraries", "Discrete Mathematics", "Distributed, Parallel, and Cluster Computing", "Emerging Technologies", "Formal Languages and Automata Theory", "General Literature", "Graphics", "Hardware Architecture", "Human-Computer Interaction", "Information Retrieval", "Information Theory", "Logic in Computer Science", "Machine Learning", "Mathematical Software", "Multiagent Systems", "Multimedia", "Networking and Internet Architecture", "Neural and Evolutionary Computing", "Numerical Analysis", "Operating Systems", "Other Computer Science", "Performance", "Programming Languages", "Robotics", "Social and Information Networks", "Software Engineering", "Sound", "Symbolic Computation", "Systems and Control"],
- "Quantitative Biology": ["Biomolecules", "Cell Behavior", "Genomics", "Molecular Networks", "Neurons and Cognition", "Other Quantitative Biology", "Populations and Evolution", "Quantitative Methods", "Subcellular Processes", "Tissues and Organs"],
- "Quantitative Finance": ["Computational Finance", "Economics", "General Finance", "Mathematical Finance", "Portfolio Management", "Pricing of Securities", "Risk Management", "Statistical Finance", "Trading and Market Microstructure"],
- "Statistics": ["Applications", "Computation", "Machine Learning", "Methodology", "Other Statistics", "Statistics Theory"],
- "Electrical Engineering and Systems Science": ["Audio and Speech Processing", "Image and Video Processing", "Signal Processing", "Systems and Control"],
- "Economics": ["Econometrics", "General Economics", "Theoretical Economics"]
-}
-
-
-def sample(email, topic, physics_topic, categories, interest):
- if not topic:
- raise gr.Error("You must choose a topic.")
- if topic == "Physics":
- if isinstance(physics_topic, list):
- raise gr.Error("You must choose a physics topic.")
- topic = physics_topic
- abbr = physics_topics[topic]
- else:
- abbr = topics[topic]
- if categories:
- papers = get_papers(abbr)
- papers = [
- t for t in papers
- if bool(set(process_subject_fields(t['subjects'])) & set(categories))][:4]
- else:
- papers = get_papers(abbr, limit=4)
- if interest:
- if not openai.api_key: raise gr.Error("Set your OpenAI api key on the left first")
- relevancy, _ = generate_relevance_score(
- papers,
- query={"interest": interest},
- threshold_score=0,
- num_paper_in_prompt=4)
- return "\n\n".join([paper["summarized_text"] for paper in relevancy])
- else:
- return "\n\n".join(f"Title: {paper['title']}\nAuthors: {paper['authors']}" for paper in papers)
-
-
-def change_subsubject(subject, physics_subject):
- if subject != "Physics":
- return gr.Dropdown.update(choices=categories_map[subject], value=[], visible=True)
- else:
- if physics_subject and not isinstance(physics_subject, list):
- return gr.Dropdown.update(choices=categories_map[physics_subject], value=[], visible=True)
- else:
- return gr.Dropdown.update(choices=[], value=[], visible=False)
-
-
-def change_physics(subject):
- if subject != "Physics":
- return gr.Dropdown.update(visible=False, value=[])
- else:
- return gr.Dropdown.update(physics_topics, visible=True)
-
-
-def test(email, topic, physics_topic, categories, interest, key):
- if not email: raise gr.Error("Set your email")
- if not key: raise gr.Error("Set your SendGrid key")
- if topic == "Physics":
- if isinstance(physics_topic, list):
- raise gr.Error("You must choose a physics topic.")
- topic = physics_topic
- abbr = physics_topics[topic]
- else:
- abbr = topics[topic]
- if categories:
- papers = get_papers(abbr)
- papers = [
- t for t in papers
- if bool(set(process_subject_fields(t['subjects'])) & set(categories))][:4]
- else:
- papers = get_papers(abbr, limit=4)
- if interest:
- if not openai.api_key: raise gr.Error("Set your OpenAI api key on the left first")
- relevancy, hallucination = generate_relevance_score(
- papers,
- query={"interest": interest},
- threshold_score=7,
- num_paper_in_prompt=8)
- body = "
".join([f'Title: {paper["title"]}
Authors: {paper["authors"]}
Score: {paper["Relevancy score"]}
Reason: {paper["Reasons for match"]}' for paper in relevancy])
- if hallucination:
- body = "Warning: the model hallucinated some papers. We have tried to remove them, but the scores may not be accurate.
" + body
- else:
- body = "
".join([f'Title: {paper["title"]}
Authors: {paper["authors"]}' for paper in papers])
- sg = sendgrid.SendGridAPIClient(api_key=key)
- from_email = Email(email)
- to_email = To(email)
- subject = "arXiv digest"
- content = Content("text/html", body)
- mail = Mail(from_email, to_email, subject, content)
- mail_json = mail.get()
-
- # Send an HTTP POST request to /mail/send
- response = sg.client.mail.send.post(request_body=mail_json)
- if response.status_code >= 200 and response.status_code <= 300:
- return "Success!"
- else:
- return "Failure: ({response.status_code})"
-
-
-def register_openai_token(token):
- openai.api_key = token
-
-with gr.Blocks() as demo:
- with gr.Row():
- with gr.Column(scale=1):
- token = gr.Textbox(label="OpenAI API Key", type="password")
- subject = gr.Radio(
- list(topics.keys()), label="Topic"
- )
- physics_subject = gr.Dropdown(physics_topics, value=[], multiselect=False, label="Physics category", visible=False, info="")
- subsubject = gr.Dropdown(
- [], value=[], multiselect=True, label="Subtopic", info="Optional. Leaving it empty will use all subtopics.", visible=False)
- subject.change(fn=change_physics, inputs=[subject], outputs=physics_subject)
- subject.change(fn=change_subsubject, inputs=[subject, physics_subject], outputs=subsubject)
- physics_subject.change(fn=change_subsubject, inputs=[subject, physics_subject], outputs=subsubject)
-
- interest = gr.Textbox(label="A natural language description of what you are interested in. We will generate relevancy scores (1-10) and explanations for the papers in the selected topics according to this statement.", info="Press shift-enter or click the button below to update.", lines=7)
- sample_btn = gr.Button("Generate Digest")
- sample_output = gr.Textbox(label="Results for your configuration.", info="For runtime purposes, this is only done on a small subset of recent papers in the topic you have selected. Papers will not be filtered by relevancy, only sorted on a scale of 1-10.")
- with gr.Column(scale=0.40):
- with gr.Box():
- title = gr.Markdown(
- """
- # Email Setup, Optional
- Send an email to the below address using the configuration on the right. Requires a sendgrid token. These values are not needed to use the right side of this page.
-
- To create a scheduled job for this, see our [Github Repository](https://github.com/AutoLLM/ArxivDigest)
- """,
- interactive=False, show_label=False)
- email = gr.Textbox(label="Email address", type="email", placeholder="")
- sendgrid_token = gr.Textbox(label="SendGrid API Key", type="password")
- with gr.Row():
- test_btn = gr.Button("Send email")
- output = gr.Textbox(show_label=False, placeholder="email status")
- test_btn.click(fn=test, inputs=[email, subject, physics_subject, subsubject, interest, sendgrid_token], outputs=output)
- token.change(fn=register_openai_token, inputs=[token])
- sample_btn.click(fn=sample, inputs=[email, subject, physics_subject, subsubject, interest], outputs=sample_output)
- subject.change(fn=sample, inputs=[email, subject, physics_subject, subsubject, interest], outputs=sample_output)
- physics_subject.change(fn=sample, inputs=[email, subject, physics_subject, subsubject, interest], outputs=sample_output)
- subsubject.change(fn=sample, inputs=[email, subject, physics_subject, subsubject, interest], outputs=sample_output)
- interest.submit(fn=sample, inputs=[email, subject, physics_subject, subsubject, interest], outputs=sample_output)
-
-demo.launch(show_api=False)
diff --git a/src/app_new.py b/src/app_new.py
new file mode 100755
index 0000000..08db449
--- /dev/null
+++ b/src/app_new.py
@@ -0,0 +1,1101 @@
+import gradio as gr
+from download_new_papers import get_papers
+import utils
+from relevancy import generate_relevance_score, process_subject_fields
+
+import os
+import openai
+import datetime
+import yaml
+from paths import DATA_DIR, DIGEST_DIR
+from model_manager import model_manager, ModelProvider
+from gemini_utils import setup_gemini_api, get_topic_clustering
+
+# Load config file
+def load_config():
+ config_path = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "config.yaml")
+ try:
+ with open(config_path, 'r') as file:
+ return yaml.safe_load(file)
+ except Exception as e:
+ print(f"Error loading config: {e}")
+ return {"threshold": 2} # Default threshold if config loading fails
+
+config = load_config()
+
+# Helper function to filter papers by threshold
+def filter_papers_by_threshold(papers, threshold):
+ """Filter papers by relevancy score threshold"""
+ print(f"\n===== FILTERING PAPERS =====")
+ print(f"Only showing papers with relevancy score >= {threshold}")
+ print(f"(Change this value in config.yaml if needed)")
+
+ # Debug the paper scores
+ for i, paper in enumerate(papers):
+ print(f"Paper {i+1} - Title: {paper.get('title', 'No title')}")
+ print(f" - Score: {paper.get('Relevancy score', 'No score')}")
+ print(f" - Fields: {list(paper.keys())}")
+
+ # First extract data from gemini_analysis if it exists and Relevancy score doesn't
+ for paper in papers:
+ if "gemini_analysis" in paper and "Relevancy score" not in paper:
+ print(f"Extracting analysis data for paper: {paper.get('title')}")
+ gemini_data = paper["gemini_analysis"]
+
+ # Map Gemini analysis fields to expected fields
+ field_mapping = {
+ "relevance_score": "Relevancy score",
+ "relationship_score": "Relevancy score",
+ "paper_relevance": "Relevancy score",
+ "paper's_relationship_to_the_user's_interests": "Relevancy score",
+ "key_innovations": "Key innovations",
+ "critical_analysis": "Critical analysis",
+ "methodology_summary": "Methodology",
+ "technical_significance": "Critical analysis",
+ "related_research": "Related work"
+ }
+
+ # Copy fields using mapping
+ for gemini_field, expected_field in field_mapping.items():
+ if gemini_field in gemini_data:
+ paper[expected_field] = gemini_data[gemini_field]
+ print(f" - Mapped {gemini_field} to {expected_field}")
+
+ # If we have no score yet, look for a number in other fields
+ if "Relevancy score" not in paper:
+ # Try to find a relevance score in any field
+ for field, value in gemini_data.items():
+ if isinstance(value, (int, float)) and 1 <= value <= 10:
+ paper["Relevancy score"] = value
+ print(f" - Found score {value} in field {field}")
+ break
+ elif isinstance(value, str) and "score" in field.lower():
+ try:
+ # Try to extract a number from the string
+ import re
+ numbers = re.findall(r'\d+', value)
+ if numbers:
+ score = int(numbers[0])
+ if 1 <= score <= 10: # Validate score range
+ paper["Relevancy score"] = score
+ print(f" - Extracted score {score} from {field}: {value}")
+ break
+ except:
+ pass
+
+ # If still no score, default to threshold to include paper
+ if "Relevancy score" not in paper:
+ paper["Relevancy score"] = threshold
+ print(f" - Assigned default score {threshold}")
+
+ # Add some reasonable defaults for missing fields
+ if "Reasons for match" not in paper and "topic_classification" in gemini_data:
+ paper["Reasons for match"] = gemini_data.get("topic_classification", "Not provided")
+
+ # Set missing fields with default values
+ for field in ["Key innovations", "Critical analysis", "Goal", "Data", "Methodology",
+ "Implementation details", "Experiments & Results", "Discussion & Next steps",
+ "Related work", "Practical applications", "Key takeaways"]:
+ if field not in paper:
+ paper[field] = "Not available in analysis"
+
+ # Ensure scores are properly parsed to integers
+ for paper in papers:
+ if "Relevancy score" in paper and not isinstance(paper["Relevancy score"], int):
+ try:
+ if isinstance(paper["Relevancy score"], str) and "/" in paper["Relevancy score"]:
+ paper["Relevancy score"] = int(paper["Relevancy score"].split("/")[0])
+ else:
+ paper["Relevancy score"] = int(paper["Relevancy score"])
+ except (ValueError, TypeError):
+ print(f"WARNING: Could not convert score '{paper.get('Relevancy score')}' to integer for paper: {paper.get('title')}")
+ paper["Relevancy score"] = threshold # Use threshold as default
+
+ # Make sure all papers have required fields
+ required_fields = [
+ "Relevancy score", "Reasons for match", "Key innovations", "Critical analysis",
+ "Goal", "Data", "Methodology", "Implementation details", "Experiments & Results",
+ "Git", "Discussion & Next steps", "Related work", "Practical applications",
+ "Key takeaways"
+ ]
+
+ for paper in papers:
+ # Make sure it has a relevancy score
+ if "Relevancy score" not in paper:
+ paper["Relevancy score"] = threshold
+ print(f"Assigned default threshold score to paper: {paper.get('title')}")
+
+ # Add missing fields with default values - always ensure all fields exist
+ for field in required_fields:
+ if field not in paper or paper[field] is None:
+ paper[field] = f"Not available in the paper content"
+ print(f"Added missing field {field} to paper: {paper.get('title')}")
+ elif isinstance(paper[field], str) and (not paper[field].strip() or
+ paper[field] == "Not provided" or paper[field] == "Not available in analysis"):
+ paper[field] = f"Not available in the paper content"
+ print(f"Replaced placeholder for field {field} in paper: {paper.get('title')}")
+
+ # Now filter papers
+ filtered_papers = [p for p in papers if p.get("Relevancy score", 0) >= threshold]
+ print(f"After filtering: {len(filtered_papers)} papers remain out of {len(papers)}")
+
+ # If fewer than 10 papers passed the filter, add the highest-scoring papers below threshold
+ # This ensures we always show a reasonable number of papers
+ if len(filtered_papers) < 10 and len(papers) > len(filtered_papers):
+ print(f"WARNING: Only {len(filtered_papers)} papers passed the threshold filter. Adding more papers.")
+ # Sort remaining papers by score and add the highest scoring ones
+ remaining_papers = [p for p in papers if p not in filtered_papers]
+ remaining_papers.sort(key=lambda x: x.get("Relevancy score", 0), reverse=True)
+ # Add enough papers to get to 10 or all remaining papers, whichever is less
+ additional_count = min(10 - len(filtered_papers), len(remaining_papers))
+ filtered_papers.extend(remaining_papers[:additional_count])
+ print(f"Added {additional_count} additional papers below threshold. Total papers: {len(filtered_papers)}")
+
+ # Fallback if no papers passed the filter
+ if len(filtered_papers) == 0 and len(papers) > 0:
+ print("WARNING: No papers passed the threshold filter. Using all papers.")
+ filtered_papers = papers
+
+ return filtered_papers
+from design_automation import (
+ is_design_automation_paper,
+ categorize_design_paper,
+ analyze_design_techniques,
+ extract_design_metrics,
+ get_related_design_papers,
+ create_design_analysis_prompt
+)
+
+topics = {
+ "Physics": "",
+ "Mathematics": "math",
+ "Computer Science": "cs",
+ "Quantitative Biology": "q-bio",
+ "Quantitative Finance": "q-fin",
+ "Statistics": "stat",
+ "Electrical Engineering and Systems Science": "eess",
+ "Economics": "econ"
+}
+
+physics_topics = {
+ "Astrophysics": "astro-ph",
+ "Condensed Matter": "cond-mat",
+ "General Relativity and Quantum Cosmology": "gr-qc",
+ "High Energy Physics - Experiment": "hep-ex",
+ "High Energy Physics - Lattice": "hep-lat",
+ "High Energy Physics - Phenomenology": "hep-ph",
+ "High Energy Physics - Theory": "hep-th",
+ "Mathematical Physics": "math-ph",
+ "Nonlinear Sciences": "nlin",
+ "Nuclear Experiment": "nucl-ex",
+ "Nuclear Theory": "nucl-th",
+ "Physics": "physics",
+ "Quantum Physics": "quant-ph"
+}
+
+categories_map = {
+ "Astrophysics": ["Astrophysics of Galaxies", "Cosmology and Nongalactic Astrophysics", "Earth and Planetary Astrophysics", "High Energy Astrophysical Phenomena", "Instrumentation and Methods for Astrophysics", "Solar and Stellar Astrophysics"],
+ "Condensed Matter": ["Disordered Systems and Neural Networks", "Materials Science", "Mesoscale and Nanoscale Physics", "Other Condensed Matter", "Quantum Gases", "Soft Condensed Matter", "Statistical Mechanics", "Strongly Correlated Electrons", "Superconductivity"],
+ "General Relativity and Quantum Cosmology": ["None"],
+ "High Energy Physics - Experiment": ["None"],
+ "High Energy Physics - Lattice": ["None"],
+ "High Energy Physics - Phenomenology": ["None"],
+ "High Energy Physics - Theory": ["None"],
+ "Mathematical Physics": ["None"],
+ "Nonlinear Sciences": ["Adaptation and Self-Organizing Systems", "Cellular Automata and Lattice Gases", "Chaotic Dynamics", "Exactly Solvable and Integrable Systems", "Pattern Formation and Solitons"],
+ "Nuclear Experiment": ["None"],
+ "Nuclear Theory": ["None"],
+ "Physics": ["Accelerator Physics", "Applied Physics", "Atmospheric and Oceanic Physics", "Atomic and Molecular Clusters", "Atomic Physics", "Biological Physics", "Chemical Physics", "Classical Physics", "Computational Physics", "Data Analysis, Statistics and Probability", "Fluid Dynamics", "General Physics", "Geophysics", "History and Philosophy of Physics", "Instrumentation and Detectors", "Medical Physics", "Optics", "Physics and Society", "Physics Education", "Plasma Physics", "Popular Physics", "Space Physics"],
+ "Quantum Physics": ["None"],
+ "Mathematics": ["Algebraic Geometry", "Algebraic Topology", "Analysis of PDEs", "Category Theory", "Classical Analysis and ODEs", "Combinatorics", "Commutative Algebra", "Complex Variables", "Differential Geometry", "Dynamical Systems", "Functional Analysis", "General Mathematics", "General Topology", "Geometric Topology", "Group Theory", "History and Overview", "Information Theory", "K-Theory and Homology", "Logic", "Mathematical Physics", "Metric Geometry", "Number Theory", "Numerical Analysis", "Operator Algebras", "Optimization and Control", "Probability", "Quantum Algebra", "Representation Theory", "Rings and Algebras", "Spectral Theory", "Statistics Theory", "Symplectic Geometry"],
+ "Computer Science": ["Artificial Intelligence", "Computation and Language", "Computational Complexity", "Computational Engineering, Finance, and Science", "Computational Geometry", "Computer Science and Game Theory", "Computer Vision and Pattern Recognition", "Computers and Society", "Cryptography and Security", "Data Structures and Algorithms", "Databases", "Digital Libraries", "Discrete Mathematics", "Distributed, Parallel, and Cluster Computing", "Emerging Technologies", "Formal Languages and Automata Theory", "General Literature", "Graphics", "Hardware Architecture", "Human-Computer Interaction", "Information Retrieval", "Information Theory", "Logic in Computer Science", "Machine Learning", "Mathematical Software", "Multiagent Systems", "Multimedia", "Networking and Internet Architecture", "Neural and Evolutionary Computing", "Numerical Analysis", "Operating Systems", "Other Computer Science", "Performance", "Programming Languages", "Robotics", "Social and Information Networks", "Software Engineering", "Sound", "Symbolic Computation", "Systems and Control"],
+ "Quantitative Biology": ["Biomolecules", "Cell Behavior", "Genomics", "Molecular Networks", "Neurons and Cognition", "Other Quantitative Biology", "Populations and Evolution", "Quantitative Methods", "Subcellular Processes", "Tissues and Organs"],
+ "Quantitative Finance": ["Computational Finance", "Economics", "General Finance", "Mathematical Finance", "Portfolio Management", "Pricing of Securities", "Risk Management", "Statistical Finance", "Trading and Market Microstructure"],
+ "Statistics": ["Applications", "Computation", "Machine Learning", "Methodology", "Other Statistics", "Statistics Theory"],
+ "Electrical Engineering and Systems Science": ["Audio and Speech Processing", "Image and Video Processing", "Signal Processing", "Systems and Control"],
+ "Economics": ["Econometrics", "General Economics", "Theoretical Economics"]
+}
+
+
+def generate_html_report(papers, title="ArXiv Digest Results", topic=None, category=None, query=None):
+ """Generate an HTML report for the papers and save to file.
+
+ Args:
+ papers: List of paper dictionaries
+ title: Title for the HTML report
+ topic: Optional topic name for filename
+ category: Optional category name for filename
+ query: Optional dictionary with interest field for research interests
+
+ Returns:
+ Path to the HTML file
+ """
+ # Debug: Log what fields are available in each paper
+ print(f"Generating HTML report for {len(papers)} papers")
+ for i, paper in enumerate(papers):
+ print(f"Paper {i+1} fields: {list(paper.keys())}")
+ if "Key innovations" in paper:
+ print(f"Paper {i+1} has Key innovations: {paper['Key innovations'][:50]}...")
+ if "Critical analysis" in paper:
+ print(f"Paper {i+1} has Critical analysis: {paper['Critical analysis'][:50]}...")
+
+ # Create a date for the filename (without time)
+ date = datetime.datetime.now().strftime("%Y%m%d")
+
+ # Create filename with topic if provided
+ if topic:
+ # Clean up topic name for filename (remove spaces, etc.)
+ topic_clean = topic.lower().replace(" ", "_").replace("/", "_")
+ html_file = os.path.join(DIGEST_DIR, f"arxiv_digest_{topic_clean}_{date}.html")
+ else:
+ html_file = os.path.join(DIGEST_DIR, f"arxiv_digest_{date}.html")
+
+ html = f"""
+
+
+
+
+ {title}
+
+
+
+ {title}
+
+
Generated on {datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")}
+
Found {len(papers)} papers
+
Topics: {topic or "All"}
+
Threshold: {query.get("threshold", config.get("threshold", "Not specified"))}
+
+
Research Interests:
+
{query.get('interest', 'Not specified')}
+
+
No papers found matching your criteria
+
+
No papers met the relevancy threshold criteria. You can:
+
+ - Lower the threshold using the slider in Advanced Settings (currently set to {threshold})
+ - Try different research interests
+ - Check different categories or topics
+
+
+
+
+
{paper.get("authors", "Unknown authors")}
+
Subject: {paper.get("subjects", "N/A")}
+ """
+
+ # Add relevancy score and reasons if available
+ if "Relevancy score" in paper:
+ html += f'
Relevancy Score: {paper.get("Relevancy score", "N/A")}
'
+
+ if "Reasons for match" in paper:
+ html += f'
Reason for Relevance: {paper.get("Reasons for match", "")}
'
+
+ # Add design information if available
+ if "design_category" in paper or "design_techniques" in paper:
+ html += '
'
+ if "design_category" in paper:
+ html += f'
Design Category: {paper.get("design_category", "")}
'
+ if "design_techniques" in paper:
+ html += f'
Design Techniques: {", ".join(paper.get("design_techniques", []))}
'
+ html += '
'
+
+ # Add abstract
+ if "abstract" in paper:
+ html += f'
Abstract: {paper.get("abstract", "")}
'
+
+ # Helper function to format field content properly
+ def format_field_content(content):
+ if isinstance(content, list):
+ # Format list items with bullet points
+ return '
' + ''.join([f'- {item}
' for item in content]) + '
'
+ else:
+ return content
+
+ # Add key innovations and critical analysis with special styling
+ if "Key innovations" in paper:
+ formatted_innovations = format_field_content(paper.get("Key innovations", ""))
+ html += f'
Key Innovations:
{formatted_innovations}
'
+ print(f"Added Key innovations for paper {i+1}")
+ else:
+ print(f"Paper {i+1} is missing Key innovations field")
+
+ if "Critical analysis" in paper:
+ formatted_analysis = format_field_content(paper.get("Critical analysis", ""))
+ html += f'
Critical Analysis:
{formatted_analysis}
'
+ print(f"Added Critical analysis for paper {i+1}")
+ else:
+ print(f"Paper {i+1} is missing Critical analysis field")
+
+ # Add goal
+ if "Goal" in paper:
+ formatted_goal = format_field_content(paper.get("Goal", ""))
+ html += f'
'
+ print(f"Added Goal for paper {i+1}")
+ else:
+ print(f"Paper {i+1} is missing Goal field")
+
+ # Add Data
+ if "Data" in paper:
+ formatted_data = format_field_content(paper.get("Data", ""))
+ html += f'
'
+ print(f"Added Data for paper {i+1}")
+ else:
+ print(f"Paper {i+1} is missing Data field")
+
+ # Add Methodology
+ if "Methodology" in paper:
+ formatted_methodology = format_field_content(paper.get("Methodology", ""))
+ html += f'
Methodology:
{formatted_methodology}
'
+ print(f"Added Methodology for paper {i+1}")
+ else:
+ print(f"Paper {i+1} is missing Methodology field")
+
+ # Add implementation details
+ if "Implementation details" in paper:
+ formatted_details = format_field_content(paper.get("Implementation details", ""))
+ html += f'
Implementation Details:
{formatted_details}
'
+ print(f"Added Implementation details for paper {i+1}")
+ else:
+ print(f"Paper {i+1} is missing Implementation details field")
+
+ # Add experiments and results
+ if "Experiments & Results" in paper:
+ formatted_results = format_field_content(paper.get("Experiments & Results", ""))
+ html += f'
Experiments & Results:
{formatted_results}
'
+ print(f"Added Experiments & Results for paper {i+1}")
+ else:
+ print(f"Paper {i+1} is missing Experiments & Results field")
+
+ # Add discussion and next steps
+ if "Discussion & Next steps" in paper:
+ formatted_discussion = format_field_content(paper.get("Discussion & Next steps", ""))
+ html += f'
Discussion & Next Steps:
{formatted_discussion}
'
+ print(f"Added Discussion & Next steps for paper {i+1}")
+ else:
+ print(f"Paper {i+1} is missing Discussion & Next steps field")
+
+ # Add Related work
+ if "Related work" in paper:
+ formatted_related = format_field_content(paper.get("Related work", ""))
+ html += f'
Related Work:
{formatted_related}
'
+ print(f"Added Related work for paper {i+1}")
+ else:
+ print(f"Paper {i+1} is missing Related work field")
+
+ # Add Practical applications
+ if "Practical applications" in paper:
+ formatted_applications = format_field_content(paper.get("Practical applications", ""))
+ html += f'
Practical Applications:
{formatted_applications}
'
+ print(f"Added Practical applications for paper {i+1}")
+ else:
+ print(f"Paper {i+1} is missing Practical applications field")
+
+ # Add Key takeaways
+ if "Key takeaways" in paper:
+ formatted_takeaways = format_field_content(paper.get("Key takeaways", ""))
+ html += f'
Key Takeaways:
{formatted_takeaways}
'
+ print(f"Added Key takeaways for paper {i+1}")
+ else:
+ print(f"Paper {i+1} is missing Key takeaways field")
+
+ # Add remaining sections that weren't already handled specifically above
+ for key, value in paper.items():
+ # Skip fields we've already handled or don't want to display
+ if key in ["title", "authors", "subjects", "main_page", "Relevancy score", "Reasons for match",
+ "design_category", "design_techniques", "summarized_text", "abstract", "content",
+ "Key innovations", "Critical analysis", "Goal", "Data", "Methodology",
+ "Implementation details", "Experiments & Results", "Discussion & Next steps",
+ "Related work", "Practical applications", "Key takeaways"]:
+ continue
+
+ if isinstance(value, str) and value.strip():
+ # Choose appropriate styling based on the section
+ section_class = "section"
+
+ if "goal" in key.lower() or "aim" in key.lower():
+ section_class = "key-section"
+ elif "data" in key.lower() or "methodology" in key.lower():
+ section_class = "implementation"
+ elif "related" in key.lower() or "practical" in key.lower() or "takeaway" in key.lower():
+ section_class = "discussion"
+
+ formatted_value = format_field_content(value)
+ html += f'
'
+ print(f"Added additional field {key} for paper {i+1}")
+
+ # Add links
+ html += f"""
+
+
Design Automation Papers
+
+
+
Found {len(self.papers)} papers related to graphic design automation with AI/ML
+
Generated on {datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")}
+
Keywords: {', '.join(DESIGN_AUTOMATION_KEYWORDS[:5])}...
+
Summary Statistics
'
+
+ html += "
Categories:
"
+ for category, count in sorted(categories.items(), key=lambda x: x[1], reverse=True):
+ html += f"- {category}: {count} papers
"
+ html += "
"
+
+ html += "
Techniques:
"
+ for technique, count in sorted(techniques.items(), key=lambda x: x[1], reverse=True):
+ html += f"- {technique}: {count} papers
"
+ html += "
Papers
'
+ for paper in self.papers:
+ html += f"""
+
+
+
{paper.get("authors", "Unknown authors")}
+
Category: {paper.get("design_category", "General")} | Subject: {paper.get("subjects", "N/A")}
+
Techniques: {', '.join(paper.get("design_techniques", ["None identified"]))}
+
Abstract: {paper.get("abstract", "No abstract available")}
+
+
Design Automation Papers
+
+
Found {len(papers)} papers related to graphic design automation with AI/ML
+
Generated on {datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")}
+
Summary Statistics
"
+
+ html += "
Categories:
"
+ for category, count in sorted(categories.items(), key=lambda x: x[1], reverse=True):
+ html += f"- {category}: {count} papers
"
+ html += "
"
+
+ html += "
Techniques:
"
+ for technique, count in sorted(techniques.items(), key=lambda x: x[1], reverse=True):
+ html += f"- {technique}: {count} papers
"
+ html += "
+
+
{paper.get("authors", "Unknown authors")}
+
Category: {paper.get("design_category", "General")} | Subject: {paper.get("subjects", "N/A")}
+
Techniques: {', '.join(paper.get("design_techniques", ["None identified"]))}
+ """
+
+ # Add relevancy score and reasons if available
+ if "Relevancy score" in paper:
+ html += f'
Relevancy Score: {paper.get("Relevancy score", "N/A")}
'
+
+ if "Reasons for match" in paper:
+ html += f'
Reason: {paper.get("Reasons for match", "")}
'
+
+ # Add abstract
+ if "abstract" in paper:
+ html += f'
Abstract: {paper.get("abstract", "")}
'
+
+ # Add all the additional analysis sections
+ for key, value in paper.items():
+ if key in ["title", "authors", "subjects", "main_page", "Relevancy score", "Reasons for match",
+ "design_category", "design_techniques", "content", "abstract"]:
+ continue
+
+ if isinstance(value, str) and value.strip():
+ html += f'
'
+
+ # Add links
+ html += f"""
+
+
Design Automation Papers
+
+
Found {len(papers)} papers related to graphic design automation with AI/ML
+
Generated on {datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")}
+
Summary Statistics
"
+
+ html += "
Categories:
"
+ for category, count in sorted(categories.items(), key=lambda x: x[1], reverse=True):
+ html += f"- {category}: {count} papers
"
+ html += "
"
+
+ html += "
Techniques:
"
+ for technique, count in sorted(techniques.items(), key=lambda x: x[1], reverse=True):
+ html += f"- {technique}: {count} papers
"
+ html += "
+
+
{paper.get("authors", "Unknown authors")}
+
arXiv ID: {paper.get("paper_id", "Unknown")}
+
Category: {paper.get("design_category", "General")} | Subject: {paper.get("subjects", "N/A")}
+
Techniques: {', '.join(paper.get("design_techniques", ["None identified"]))}
+
Evaluation metrics: {', '.join(paper.get("design_metrics", ["None identified"]))}
+
Abstract: {paper.get("abstract", "No abstract available")}
+