wc clone does not support multibyte character

Question

I'm trying to complete a coding challenge by cloning the wc unix tool to rust. I have a challenge with the final step that is to support the command line option -m that outputs the number of characters in a file. If the current locale does not support multibyte characters this will match the -c option.

Here is a link to the challenge description: https://codingchallenges.fyi/challenges/challenge-wc/

The implemented option outputs less than the expected number due to locale issues perhaps. The expected output is 339292 test.txt while I get 327900.

I have tried to wrap my head around unicode but I still dont get it. https://learn.microsoft.com/en-us/globalization/locale/locale and https://tonsky.me/blog/unicode/

The testing file is available at https://github.com/andwati/wc-rs/blob/main/test.txt

Here is my implementation. I am very new to Rust so the code might not be idiomatic.

use std::env;
use std::fs::File;
use std::io::prelude::*;
use std::io::{self, BufReader};

fn number_of_bytes(file_path: &str) -> io::Result<()> {
    let f = File::open(file_path)?;

    let mut reader = BufReader::new(f);
    let mut buffer = Vec::new();

    // read the whole file
    reader.read_to_end(&mut buffer)?;

    let total_bytes = buffer.len();
    println!("{} {}", total_bytes, file_path);
    Ok(())
}

fn number_of_lines(file_path: &str) -> io::Result<()> {
    let f = File::open(file_path)?;
    let reader = BufReader::new(f);

    let line_count = reader.lines().count();
    println!("{} {}", line_count, file_path);

    Ok(())
}
fn number_of_words(file_path: &str) {
    let f = File::open(file_path).expect("Error opening the file");
    let reader = BufReader::new(f);

    let mut word_count: u32 = 0;
    for line in reader.lines() {
        let curr: String = line.expect("Error reading content of the file");
        // let words: Vec<&str> = curr.split(" ").collect();
        let words: Vec<&str> = curr.split_whitespace().collect();
        let filtered_words: Vec<&str> = words.into_iter().filter(|word| word.len() > 0).collect();
        word_count += filtered_words.len() as u32
    }

    println!("{}", word_count);
}

fn number_of_characters(file_path: &str) {
    let mut file = File::open(file_path).unwrap();
    let mut s = String::new();
    file.read_to_string(&mut s).unwrap();
    print!("{}", s.trim_end().chars().count());
}

fn main() {
    let args: Vec<String> = env::args().collect();
    let file_path = &args[2];

    if args.len() > 1 && args[1] == "-c" {
        number_of_bytes(file_path).unwrap();
    } else if args.len() > 1 && args[1] == "-l" {
        number_of_lines(file_path).unwrap();
    } else if args.len() > 1 && args[1] == "-w" {
        number_of_words(&file_path);
    } else if args.len() > 1 && args[1] == "-m" {
        number_of_characters(file_path);
    } else {
        eprintln!("Usage: wc-tool -c <filepath>");
        std::process::exit(1);
    }
}

I have tried to wrap my head around unicode but I still dont get it. https://learn.microsoft.com/en-us/globalization/locale/locale and https://tonsky.me/blog/unicode/ .

Answer 1

I was able to get accurate readings by implementing the function to

fn number_of_characters(file_path: &str) {
    let mut file = File::open(file_path).unwrap();
    let mut s = String::new();
    file.read_to_string(&mut s).unwrap();
    print!("{}", s.chars().count());
}