Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................X....................] duration: 3.386952004s trace: https://jaeger.dev.storj.io/trace/5cda11e56fceed5a FAIL (2025-12-14 10:06:41 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.386978895s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................X...................] duration: 3.54085975s trace: https://jaeger.dev.storj.io/trace/5cda11e56fceed5a FAIL (2025-12-14 10:06:23 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.540892652s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................X..................] duration: 3.675969211s trace: https://jaeger.dev.storj.io/trace/5cda11e56fceed5a FAIL (2025-12-14 10:05:23 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.67599534s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................X.................] duration: 4.616607443s trace: https://jaeger.dev.storj.io/trace/5cda11e56fceed5a FAIL (2025-12-14 10:04:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.616632981s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................X................] duration: 3.981852376s trace: https://jaeger.dev.storj.io/trace/5cda11e56fceed5a FAIL (2025-12-14 10:03:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.982038291s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................................X...............] duration: 3.635675626s trace: https://jaeger.dev.storj.io/trace/5cda11e56fceed5a FAIL (2025-12-14 10:02:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.635703078s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................................X..............] duration: 3.608254361s trace: https://jaeger.dev.storj.io/trace/5cda11e56fceed5a FAIL (2025-12-14 10:02:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.608289939s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................................X.............] duration: 3.458569865s trace: https://jaeger.dev.storj.io/trace/5cda11e56fceed5a FAIL (2025-12-14 10:01:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.458596516s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................................X............] duration: 3.777200424s trace: https://jaeger.dev.storj.io/trace/5cda11e56fceed5a FAIL (2025-12-14 10:01:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.777229139s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................................X...........] duration: 3.564998365s trace: https://jaeger.dev.storj.io/trace/5cda11e56fceed5a FAIL (2025-12-14 10:00:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.565024174s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 3.460006069s trace: https://jaeger.dev.storj.io/trace/5cda11e56fceed5a FAIL (2025-12-14 09:59:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.460019176s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 3.456551783s trace: https://jaeger.dev.storj.io/trace/5cda11e56fceed5a FAIL (2025-12-14 09:59:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.456587671s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 3.507970063s trace: https://jaeger.dev.storj.io/trace/5cda11e56fceed5a FAIL (2025-12-14 09:58:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.507996813s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 3.465167109s trace: https://jaeger.dev.storj.io/trace/5cda11e56fceed5a FAIL (2025-12-14 09:57:12 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.465195301s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 3.485351213s trace: https://jaeger.dev.storj.io/trace/5cda11e56fceed5a FAIL (2025-12-14 09:56:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.485372823s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 3.424120156s trace: https://jaeger.dev.storj.io/trace/5cda11e56fceed5a FAIL (2025-12-14 09:55:49 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.42414849s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 3.482926303s trace: https://jaeger.dev.storj.io/trace/5cda11e56fceed5a FAIL (2025-12-14 09:55:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.48294964s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 3.531937469s trace: https://jaeger.dev.storj.io/trace/5cda11e56fceed5a FAIL (2025-12-14 09:54:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.532556039s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 5.00120821s trace: https://jaeger.dev.storj.io/trace/5cda11e56fceed5a FAIL (2025-12-14 09:53:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001233018s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 3.573090663s trace: https://jaeger.dev.storj.io/trace/5cda11e56fceed5a FAIL (2025-12-14 09:52:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.573147751s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.001414281s trace: https://jaeger.dev.storj.io/trace/5cda11e56fceed5a FAIL (2025-12-14 09:51:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001501538s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.32735181s trace: https://jaeger.dev.storj.io/trace/b3a498e2503e131 FAIL (2025-12-14 09:49:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ) context deadline exceeded; piecestore: (Node ID: 1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX, Piece ID: JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA) context deadline exceeded ======LOGS====== oad=183567]: Starting upload Dec 14 09:01:28.382436168 (209h32m28.253519203s) [upload=183567]: Done starting upload Dec 14 09:01:28.382504638 (209h32m28.253587683s) [upload=183567]: Uploading segments... Dec 14 09:01:28.383014028 (209h32m28.254097073s) [upload=183567]: Got next segment. Inline: false Dec 14 09:01:28.383065245 (209h32m28.254148281s) [upload=183567]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 09:01:28.383070747 (209h32m28.254153782s) [upload=183567]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 09:01:29.117351421 (209h32m28.988434477s) [seg_pos={0 0}, upload=183567]: Begin upload segment... Dec 14 09:01:29.117412969 (209h32m28.988496004s) [seg_pos={0 0}, upload=183567]: Done begin upload segment. Dec 14 09:01:29.117442345 (209h32m28.98852538s) [noise=true , address=145.79.227.26:4655 , piece_id=6YPJCS6WMI5AYEEESQSWTPK7X4FIUO53V2FONNQYZEKGGBO5PXBA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118002081 (209h32m28.989085116s) [noise=true , address=145.79.227.28:4725 , piece_id=ZDTVQMPD4EUWKRQQVP7PNAQ634L4MQBYJIAFPFSKLQUOTFZHMFPA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118088185 (209h32m28.98917122s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118501743 (209h32m28.989584788s) [noise=true , address=77.90.31.21:4878 , piece_id=HF6KZLDP62HJG7MWOAF4TTPXLI7C6CAUX5O4QWT5MP7ZWNZUWDDQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119009108 (209h32m28.990092143s) [noise=true , address=145.79.227.26:4653 , piece_id=RWUX47DDMWRTEXWKCDN6BF2PGSL6BZAXWQIX5DT547AKMWLNDWYQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119070265 (209h32m28.99015331s) [noise=true , address=145.79.227.47:4265 , piece_id=JLGBSCGU5PK7AAYNRKSJRLMQSKCG4SRM4WSVRALCW7GCA56LRFYQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119374144 (209h32m28.990457179s) [noise=true , address=77.90.31.14:4662 , piece_id=TQLMWNW5JAB62NZRPIMTM33N7GSAHZGCU3V7H32LZTSHE2SWMCIQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119462752 (209h32m28.990545797s) [noise=true , address=145.79.227.46:4601 , piece_id=NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119754849 (209h32m28.990837894s) [noise=true , address=145.79.227.41:4087 , piece_id=66RGE7N2TJ7PSWIBB2UDPIKNZ2ND4BZXFLEXDZWVMVMAYL5PSBGA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119815303 (209h32m28.990898338s) [noise=true , address=145.79.227.49:4337 , piece_id=HUJLCBY3PQDUJ4Y52LESV6CVZ5DMMCV4MO346NTVY3QCSG6NSZCQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120045883 (209h32m28.991128918s) [noise=true , address=145.79.227.31:4799 , piece_id=L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120254059 (209h32m28.991337094s) [noise=true , address=145.79.227.23:4546 , piece_id=3Q24KFURFMF5POQJ2MNRGWX32E6IKI6XW5XHSLTQ7CCXTEUTGW6A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120325726 (209h32m28.991408871s) [noise=true , address=145.79.227.42:4123 , piece_id=53K6JA7I4GAUEL3ROO65MKFXN7ZSNMOH22UGAXIBRW4WUKN7555A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120653820 (209h32m28.991736865s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.121122984 (209h32m28.992206029s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.121668723 (209h32m28.992751768s) [noise=true , address=145.79.227.46:4603 , piece_id=JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:30.977492703 (209h32m30.848575748s) [upload=183567]: Next returned nil segment Dec 14 09:01:30.977525565 (209h32m30.84860862s) [upload=183567]: Waiting for error group managing segments... Dec 14 09:01:31.270408579 (209h32m31.141491624s) [noise=true , address=145.79.227.42:4123 , piece_id=53K6JA7I4GAUEL3ROO65MKFXN7ZSNMOH22UGAXIBRW4WUKN7555A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.315656118 (209h32m31.186739163s) [noise=true , address=77.90.31.14:4662 , piece_id=TQLMWNW5JAB62NZRPIMTM33N7GSAHZGCU3V7H32LZTSHE2SWMCIQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.342981501 (209h32m31.214064536s) [noise=true , address=77.90.31.21:4878 , piece_id=HF6KZLDP62HJG7MWOAF4TTPXLI7C6CAUX5O4QWT5MP7ZWNZUWDDQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.739040250 (209h32m31.610123295s) [noise=true , address=145.79.227.23:4546 , piece_id=3Q24KFURFMF5POQJ2MNRGWX32E6IKI6XW5XHSLTQ7CCXTEUTGW6A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.800414969 (209h32m31.671498004s) [noise=true , address=145.79.227.47:4265 , piece_id=JLGBSCGU5PK7AAYNRKSJRLMQSKCG4SRM4WSVRALCW7GCA56LRFYQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:32.628091848 (209h32m32.499174883s) [noise=true , address=145.79.227.26:4653 , piece_id=RWUX47DDMWRTEXWKCDN6BF2PGSL6BZAXWQIX5DT547AKMWLNDWYQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:32.629981676 (209h32m32.501064721s) [noise=true , address=145.79.227.49:4337 , piece_id=HUJLCBY3PQDUJ4Y52LESV6CVZ5DMMCV4MO346NTVY3QCSG6NSZCQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.054036275 (209h32m32.92511931s) [noise=true , address=145.79.227.41:4087 , piece_id=66RGE7N2TJ7PSWIBB2UDPIKNZ2ND4BZXFLEXDZWVMVMAYL5PSBGA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.071010907 (209h32m32.942093952s) [noise=true , address=145.79.227.31:4799 , piece_id=L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.120830402 (209h32m32.991913438s) [noise=true , address=145.79.227.46:4601 , piece_id=NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:34.312983241 (209h32m34.184066276s) [noise=true , address=145.79.227.26:4655 , piece_id=6YPJCS6WMI5AYEEESQSWTPK7X4FIUO53V2FONNQYZEKGGBO5PXBA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:34.492220262 (209h32m34.363303307s) [noise=true , address=145.79.227.28:4725 , piece_id=ZDTVQMPD4EUWKRQQVP7PNAQ634L4MQBYJIAFPFSKLQUOTFZHMFPA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:35.590754704 (209h32m35.461837739s) [noise=true , address=145.79.227.46:4603 , piece_id=JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:35.590780002 (209h32m35.461863047s) [seg_pos={0 0}, upload=183567]: Segment reached optimal threshold of 13 pieces. Dec 14 09:01:35.591192537 (209h32m35.462275572s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: read tcp 188.166.225.170:47850->145.79.227.49:4339: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591202196 (209h32m35.462285231s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591227634 (209h32m35.462310669s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: rpc: dial tcp 145.79.227.39:4014: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591241200 (209h32m35.462324245s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591644728 (209h32m35.462727773s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:45550->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591655028 (209h32m35.462738073s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591681468 (209h32m35.462764503s) [seg_pos={0 0}, upload=183567]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 14 09:01:35.591768234 (209h32m35.462851279s) [upload=183567]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 14 09:01:35.591774916 (209h32m35.462857961s) [upload=183567]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 14 09:01:36.093868656 (209h32m35.964951701s) [upload=183567]: Done uploading segments... Dec 14 09:01:36.093895967 (209h32m35.964979012s) [upload=183567]: Upload finished. err: <nil> Duration: 6.327390665s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 7.678384576s trace: https://jaeger.dev.storj.io/trace/b3a498e2503e131 FAIL (2025-12-14 09:48:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ) context deadline exceeded; piecestore: (Node ID: 1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX, Piece ID: JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA) context deadline exceeded ======LOGS====== oad=183567]: Starting upload Dec 14 09:01:28.382436168 (209h32m28.253519203s) [upload=183567]: Done starting upload Dec 14 09:01:28.382504638 (209h32m28.253587683s) [upload=183567]: Uploading segments... Dec 14 09:01:28.383014028 (209h32m28.254097073s) [upload=183567]: Got next segment. Inline: false Dec 14 09:01:28.383065245 (209h32m28.254148281s) [upload=183567]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 09:01:28.383070747 (209h32m28.254153782s) [upload=183567]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 09:01:29.117351421 (209h32m28.988434477s) [seg_pos={0 0}, upload=183567]: Begin upload segment... Dec 14 09:01:29.117412969 (209h32m28.988496004s) [seg_pos={0 0}, upload=183567]: Done begin upload segment. Dec 14 09:01:29.117442345 (209h32m28.98852538s) [noise=true , address=145.79.227.26:4655 , piece_id=6YPJCS6WMI5AYEEESQSWTPK7X4FIUO53V2FONNQYZEKGGBO5PXBA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118002081 (209h32m28.989085116s) [noise=true , address=145.79.227.28:4725 , piece_id=ZDTVQMPD4EUWKRQQVP7PNAQ634L4MQBYJIAFPFSKLQUOTFZHMFPA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118088185 (209h32m28.98917122s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118501743 (209h32m28.989584788s) [noise=true , address=77.90.31.21:4878 , piece_id=HF6KZLDP62HJG7MWOAF4TTPXLI7C6CAUX5O4QWT5MP7ZWNZUWDDQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119009108 (209h32m28.990092143s) [noise=true , address=145.79.227.26:4653 , piece_id=RWUX47DDMWRTEXWKCDN6BF2PGSL6BZAXWQIX5DT547AKMWLNDWYQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119070265 (209h32m28.99015331s) [noise=true , address=145.79.227.47:4265 , piece_id=JLGBSCGU5PK7AAYNRKSJRLMQSKCG4SRM4WSVRALCW7GCA56LRFYQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119374144 (209h32m28.990457179s) [noise=true , address=77.90.31.14:4662 , piece_id=TQLMWNW5JAB62NZRPIMTM33N7GSAHZGCU3V7H32LZTSHE2SWMCIQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119462752 (209h32m28.990545797s) [noise=true , address=145.79.227.46:4601 , piece_id=NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119754849 (209h32m28.990837894s) [noise=true , address=145.79.227.41:4087 , piece_id=66RGE7N2TJ7PSWIBB2UDPIKNZ2ND4BZXFLEXDZWVMVMAYL5PSBGA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119815303 (209h32m28.990898338s) [noise=true , address=145.79.227.49:4337 , piece_id=HUJLCBY3PQDUJ4Y52LESV6CVZ5DMMCV4MO346NTVY3QCSG6NSZCQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120045883 (209h32m28.991128918s) [noise=true , address=145.79.227.31:4799 , piece_id=L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120254059 (209h32m28.991337094s) [noise=true , address=145.79.227.23:4546 , piece_id=3Q24KFURFMF5POQJ2MNRGWX32E6IKI6XW5XHSLTQ7CCXTEUTGW6A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120325726 (209h32m28.991408871s) [noise=true , address=145.79.227.42:4123 , piece_id=53K6JA7I4GAUEL3ROO65MKFXN7ZSNMOH22UGAXIBRW4WUKN7555A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120653820 (209h32m28.991736865s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.121122984 (209h32m28.992206029s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.121668723 (209h32m28.992751768s) [noise=true , address=145.79.227.46:4603 , piece_id=JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:30.977492703 (209h32m30.848575748s) [upload=183567]: Next returned nil segment Dec 14 09:01:30.977525565 (209h32m30.84860862s) [upload=183567]: Waiting for error group managing segments... Dec 14 09:01:31.270408579 (209h32m31.141491624s) [noise=true , address=145.79.227.42:4123 , piece_id=53K6JA7I4GAUEL3ROO65MKFXN7ZSNMOH22UGAXIBRW4WUKN7555A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.315656118 (209h32m31.186739163s) [noise=true , address=77.90.31.14:4662 , piece_id=TQLMWNW5JAB62NZRPIMTM33N7GSAHZGCU3V7H32LZTSHE2SWMCIQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.342981501 (209h32m31.214064536s) [noise=true , address=77.90.31.21:4878 , piece_id=HF6KZLDP62HJG7MWOAF4TTPXLI7C6CAUX5O4QWT5MP7ZWNZUWDDQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.739040250 (209h32m31.610123295s) [noise=true , address=145.79.227.23:4546 , piece_id=3Q24KFURFMF5POQJ2MNRGWX32E6IKI6XW5XHSLTQ7CCXTEUTGW6A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.800414969 (209h32m31.671498004s) [noise=true , address=145.79.227.47:4265 , piece_id=JLGBSCGU5PK7AAYNRKSJRLMQSKCG4SRM4WSVRALCW7GCA56LRFYQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:32.628091848 (209h32m32.499174883s) [noise=true , address=145.79.227.26:4653 , piece_id=RWUX47DDMWRTEXWKCDN6BF2PGSL6BZAXWQIX5DT547AKMWLNDWYQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:32.629981676 (209h32m32.501064721s) [noise=true , address=145.79.227.49:4337 , piece_id=HUJLCBY3PQDUJ4Y52LESV6CVZ5DMMCV4MO346NTVY3QCSG6NSZCQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.054036275 (209h32m32.92511931s) [noise=true , address=145.79.227.41:4087 , piece_id=66RGE7N2TJ7PSWIBB2UDPIKNZ2ND4BZXFLEXDZWVMVMAYL5PSBGA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.071010907 (209h32m32.942093952s) [noise=true , address=145.79.227.31:4799 , piece_id=L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.120830402 (209h32m32.991913438s) [noise=true , address=145.79.227.46:4601 , piece_id=NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:34.312983241 (209h32m34.184066276s) [noise=true , address=145.79.227.26:4655 , piece_id=6YPJCS6WMI5AYEEESQSWTPK7X4FIUO53V2FONNQYZEKGGBO5PXBA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:34.492220262 (209h32m34.363303307s) [noise=true , address=145.79.227.28:4725 , piece_id=ZDTVQMPD4EUWKRQQVP7PNAQ634L4MQBYJIAFPFSKLQUOTFZHMFPA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:35.590754704 (209h32m35.461837739s) [noise=true , address=145.79.227.46:4603 , piece_id=JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:35.590780002 (209h32m35.461863047s) [seg_pos={0 0}, upload=183567]: Segment reached optimal threshold of 13 pieces. Dec 14 09:01:35.591192537 (209h32m35.462275572s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: read tcp 188.166.225.170:47850->145.79.227.49:4339: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591202196 (209h32m35.462285231s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591227634 (209h32m35.462310669s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: rpc: dial tcp 145.79.227.39:4014: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591241200 (209h32m35.462324245s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591644728 (209h32m35.462727773s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:45550->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591655028 (209h32m35.462738073s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591681468 (209h32m35.462764503s) [seg_pos={0 0}, upload=183567]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 14 09:01:35.591768234 (209h32m35.462851279s) [upload=183567]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 14 09:01:35.591774916 (209h32m35.462857961s) [upload=183567]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 14 09:01:36.093868656 (209h32m35.964951701s) [upload=183567]: Done uploading segments... Dec 14 09:01:36.093895967 (209h32m35.964979012s) [upload=183567]: Upload finished. err: <nil> Duration: 7.67856029s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.XX.........................................................] duration: 5.756063597s trace: https://jaeger.dev.storj.io/trace/b3a498e2503e131 FAIL (2025-12-14 09:47:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ) context deadline exceeded; piecestore: (Node ID: 1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX, Piece ID: JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA) context deadline exceeded ======LOGS====== oad=183567]: Starting upload Dec 14 09:01:28.382436168 (209h32m28.253519203s) [upload=183567]: Done starting upload Dec 14 09:01:28.382504638 (209h32m28.253587683s) [upload=183567]: Uploading segments... Dec 14 09:01:28.383014028 (209h32m28.254097073s) [upload=183567]: Got next segment. Inline: false Dec 14 09:01:28.383065245 (209h32m28.254148281s) [upload=183567]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 09:01:28.383070747 (209h32m28.254153782s) [upload=183567]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 09:01:29.117351421 (209h32m28.988434477s) [seg_pos={0 0}, upload=183567]: Begin upload segment... Dec 14 09:01:29.117412969 (209h32m28.988496004s) [seg_pos={0 0}, upload=183567]: Done begin upload segment. Dec 14 09:01:29.117442345 (209h32m28.98852538s) [noise=true , address=145.79.227.26:4655 , piece_id=6YPJCS6WMI5AYEEESQSWTPK7X4FIUO53V2FONNQYZEKGGBO5PXBA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118002081 (209h32m28.989085116s) [noise=true , address=145.79.227.28:4725 , piece_id=ZDTVQMPD4EUWKRQQVP7PNAQ634L4MQBYJIAFPFSKLQUOTFZHMFPA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118088185 (209h32m28.98917122s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118501743 (209h32m28.989584788s) [noise=true , address=77.90.31.21:4878 , piece_id=HF6KZLDP62HJG7MWOAF4TTPXLI7C6CAUX5O4QWT5MP7ZWNZUWDDQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119009108 (209h32m28.990092143s) [noise=true , address=145.79.227.26:4653 , piece_id=RWUX47DDMWRTEXWKCDN6BF2PGSL6BZAXWQIX5DT547AKMWLNDWYQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119070265 (209h32m28.99015331s) [noise=true , address=145.79.227.47:4265 , piece_id=JLGBSCGU5PK7AAYNRKSJRLMQSKCG4SRM4WSVRALCW7GCA56LRFYQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119374144 (209h32m28.990457179s) [noise=true , address=77.90.31.14:4662 , piece_id=TQLMWNW5JAB62NZRPIMTM33N7GSAHZGCU3V7H32LZTSHE2SWMCIQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119462752 (209h32m28.990545797s) [noise=true , address=145.79.227.46:4601 , piece_id=NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119754849 (209h32m28.990837894s) [noise=true , address=145.79.227.41:4087 , piece_id=66RGE7N2TJ7PSWIBB2UDPIKNZ2ND4BZXFLEXDZWVMVMAYL5PSBGA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119815303 (209h32m28.990898338s) [noise=true , address=145.79.227.49:4337 , piece_id=HUJLCBY3PQDUJ4Y52LESV6CVZ5DMMCV4MO346NTVY3QCSG6NSZCQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120045883 (209h32m28.991128918s) [noise=true , address=145.79.227.31:4799 , piece_id=L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120254059 (209h32m28.991337094s) [noise=true , address=145.79.227.23:4546 , piece_id=3Q24KFURFMF5POQJ2MNRGWX32E6IKI6XW5XHSLTQ7CCXTEUTGW6A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120325726 (209h32m28.991408871s) [noise=true , address=145.79.227.42:4123 , piece_id=53K6JA7I4GAUEL3ROO65MKFXN7ZSNMOH22UGAXIBRW4WUKN7555A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120653820 (209h32m28.991736865s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.121122984 (209h32m28.992206029s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.121668723 (209h32m28.992751768s) [noise=true , address=145.79.227.46:4603 , piece_id=JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:30.977492703 (209h32m30.848575748s) [upload=183567]: Next returned nil segment Dec 14 09:01:30.977525565 (209h32m30.84860862s) [upload=183567]: Waiting for error group managing segments... Dec 14 09:01:31.270408579 (209h32m31.141491624s) [noise=true , address=145.79.227.42:4123 , piece_id=53K6JA7I4GAUEL3ROO65MKFXN7ZSNMOH22UGAXIBRW4WUKN7555A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.315656118 (209h32m31.186739163s) [noise=true , address=77.90.31.14:4662 , piece_id=TQLMWNW5JAB62NZRPIMTM33N7GSAHZGCU3V7H32LZTSHE2SWMCIQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.342981501 (209h32m31.214064536s) [noise=true , address=77.90.31.21:4878 , piece_id=HF6KZLDP62HJG7MWOAF4TTPXLI7C6CAUX5O4QWT5MP7ZWNZUWDDQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.739040250 (209h32m31.610123295s) [noise=true , address=145.79.227.23:4546 , piece_id=3Q24KFURFMF5POQJ2MNRGWX32E6IKI6XW5XHSLTQ7CCXTEUTGW6A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.800414969 (209h32m31.671498004s) [noise=true , address=145.79.227.47:4265 , piece_id=JLGBSCGU5PK7AAYNRKSJRLMQSKCG4SRM4WSVRALCW7GCA56LRFYQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:32.628091848 (209h32m32.499174883s) [noise=true , address=145.79.227.26:4653 , piece_id=RWUX47DDMWRTEXWKCDN6BF2PGSL6BZAXWQIX5DT547AKMWLNDWYQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:32.629981676 (209h32m32.501064721s) [noise=true , address=145.79.227.49:4337 , piece_id=HUJLCBY3PQDUJ4Y52LESV6CVZ5DMMCV4MO346NTVY3QCSG6NSZCQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.054036275 (209h32m32.92511931s) [noise=true , address=145.79.227.41:4087 , piece_id=66RGE7N2TJ7PSWIBB2UDPIKNZ2ND4BZXFLEXDZWVMVMAYL5PSBGA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.071010907 (209h32m32.942093952s) [noise=true , address=145.79.227.31:4799 , piece_id=L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.120830402 (209h32m32.991913438s) [noise=true , address=145.79.227.46:4601 , piece_id=NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:34.312983241 (209h32m34.184066276s) [noise=true , address=145.79.227.26:4655 , piece_id=6YPJCS6WMI5AYEEESQSWTPK7X4FIUO53V2FONNQYZEKGGBO5PXBA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:34.492220262 (209h32m34.363303307s) [noise=true , address=145.79.227.28:4725 , piece_id=ZDTVQMPD4EUWKRQQVP7PNAQ634L4MQBYJIAFPFSKLQUOTFZHMFPA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:35.590754704 (209h32m35.461837739s) [noise=true , address=145.79.227.46:4603 , piece_id=JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:35.590780002 (209h32m35.461863047s) [seg_pos={0 0}, upload=183567]: Segment reached optimal threshold of 13 pieces. Dec 14 09:01:35.591192537 (209h32m35.462275572s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: read tcp 188.166.225.170:47850->145.79.227.49:4339: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591202196 (209h32m35.462285231s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591227634 (209h32m35.462310669s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: rpc: dial tcp 145.79.227.39:4014: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591241200 (209h32m35.462324245s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591644728 (209h32m35.462727773s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:45550->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591655028 (209h32m35.462738073s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591681468 (209h32m35.462764503s) [seg_pos={0 0}, upload=183567]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 14 09:01:35.591768234 (209h32m35.462851279s) [upload=183567]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 14 09:01:35.591774916 (209h32m35.462857961s) [upload=183567]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 14 09:01:36.093868656 (209h32m35.964951701s) [upload=183567]: Done uploading segments... Dec 14 09:01:36.093895967 (209h32m35.964979012s) [upload=183567]: Upload finished. err: <nil> Duration: 5.756090479s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..XX........................................................] duration: 8.470178179s trace: https://jaeger.dev.storj.io/trace/b3a498e2503e131 FAIL (2025-12-14 09:46:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ) context deadline exceeded; piecestore: (Node ID: 1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX, Piece ID: JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA) context deadline exceeded ======LOGS====== oad=183567]: Starting upload Dec 14 09:01:28.382436168 (209h32m28.253519203s) [upload=183567]: Done starting upload Dec 14 09:01:28.382504638 (209h32m28.253587683s) [upload=183567]: Uploading segments... Dec 14 09:01:28.383014028 (209h32m28.254097073s) [upload=183567]: Got next segment. Inline: false Dec 14 09:01:28.383065245 (209h32m28.254148281s) [upload=183567]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 09:01:28.383070747 (209h32m28.254153782s) [upload=183567]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 09:01:29.117351421 (209h32m28.988434477s) [seg_pos={0 0}, upload=183567]: Begin upload segment... Dec 14 09:01:29.117412969 (209h32m28.988496004s) [seg_pos={0 0}, upload=183567]: Done begin upload segment. Dec 14 09:01:29.117442345 (209h32m28.98852538s) [noise=true , address=145.79.227.26:4655 , piece_id=6YPJCS6WMI5AYEEESQSWTPK7X4FIUO53V2FONNQYZEKGGBO5PXBA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118002081 (209h32m28.989085116s) [noise=true , address=145.79.227.28:4725 , piece_id=ZDTVQMPD4EUWKRQQVP7PNAQ634L4MQBYJIAFPFSKLQUOTFZHMFPA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118088185 (209h32m28.98917122s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118501743 (209h32m28.989584788s) [noise=true , address=77.90.31.21:4878 , piece_id=HF6KZLDP62HJG7MWOAF4TTPXLI7C6CAUX5O4QWT5MP7ZWNZUWDDQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119009108 (209h32m28.990092143s) [noise=true , address=145.79.227.26:4653 , piece_id=RWUX47DDMWRTEXWKCDN6BF2PGSL6BZAXWQIX5DT547AKMWLNDWYQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119070265 (209h32m28.99015331s) [noise=true , address=145.79.227.47:4265 , piece_id=JLGBSCGU5PK7AAYNRKSJRLMQSKCG4SRM4WSVRALCW7GCA56LRFYQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119374144 (209h32m28.990457179s) [noise=true , address=77.90.31.14:4662 , piece_id=TQLMWNW5JAB62NZRPIMTM33N7GSAHZGCU3V7H32LZTSHE2SWMCIQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119462752 (209h32m28.990545797s) [noise=true , address=145.79.227.46:4601 , piece_id=NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119754849 (209h32m28.990837894s) [noise=true , address=145.79.227.41:4087 , piece_id=66RGE7N2TJ7PSWIBB2UDPIKNZ2ND4BZXFLEXDZWVMVMAYL5PSBGA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119815303 (209h32m28.990898338s) [noise=true , address=145.79.227.49:4337 , piece_id=HUJLCBY3PQDUJ4Y52LESV6CVZ5DMMCV4MO346NTVY3QCSG6NSZCQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120045883 (209h32m28.991128918s) [noise=true , address=145.79.227.31:4799 , piece_id=L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120254059 (209h32m28.991337094s) [noise=true , address=145.79.227.23:4546 , piece_id=3Q24KFURFMF5POQJ2MNRGWX32E6IKI6XW5XHSLTQ7CCXTEUTGW6A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120325726 (209h32m28.991408871s) [noise=true , address=145.79.227.42:4123 , piece_id=53K6JA7I4GAUEL3ROO65MKFXN7ZSNMOH22UGAXIBRW4WUKN7555A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120653820 (209h32m28.991736865s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.121122984 (209h32m28.992206029s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.121668723 (209h32m28.992751768s) [noise=true , address=145.79.227.46:4603 , piece_id=JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:30.977492703 (209h32m30.848575748s) [upload=183567]: Next returned nil segment Dec 14 09:01:30.977525565 (209h32m30.84860862s) [upload=183567]: Waiting for error group managing segments... Dec 14 09:01:31.270408579 (209h32m31.141491624s) [noise=true , address=145.79.227.42:4123 , piece_id=53K6JA7I4GAUEL3ROO65MKFXN7ZSNMOH22UGAXIBRW4WUKN7555A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.315656118 (209h32m31.186739163s) [noise=true , address=77.90.31.14:4662 , piece_id=TQLMWNW5JAB62NZRPIMTM33N7GSAHZGCU3V7H32LZTSHE2SWMCIQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.342981501 (209h32m31.214064536s) [noise=true , address=77.90.31.21:4878 , piece_id=HF6KZLDP62HJG7MWOAF4TTPXLI7C6CAUX5O4QWT5MP7ZWNZUWDDQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.739040250 (209h32m31.610123295s) [noise=true , address=145.79.227.23:4546 , piece_id=3Q24KFURFMF5POQJ2MNRGWX32E6IKI6XW5XHSLTQ7CCXTEUTGW6A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.800414969 (209h32m31.671498004s) [noise=true , address=145.79.227.47:4265 , piece_id=JLGBSCGU5PK7AAYNRKSJRLMQSKCG4SRM4WSVRALCW7GCA56LRFYQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:32.628091848 (209h32m32.499174883s) [noise=true , address=145.79.227.26:4653 , piece_id=RWUX47DDMWRTEXWKCDN6BF2PGSL6BZAXWQIX5DT547AKMWLNDWYQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:32.629981676 (209h32m32.501064721s) [noise=true , address=145.79.227.49:4337 , piece_id=HUJLCBY3PQDUJ4Y52LESV6CVZ5DMMCV4MO346NTVY3QCSG6NSZCQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.054036275 (209h32m32.92511931s) [noise=true , address=145.79.227.41:4087 , piece_id=66RGE7N2TJ7PSWIBB2UDPIKNZ2ND4BZXFLEXDZWVMVMAYL5PSBGA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.071010907 (209h32m32.942093952s) [noise=true , address=145.79.227.31:4799 , piece_id=L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.120830402 (209h32m32.991913438s) [noise=true , address=145.79.227.46:4601 , piece_id=NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:34.312983241 (209h32m34.184066276s) [noise=true , address=145.79.227.26:4655 , piece_id=6YPJCS6WMI5AYEEESQSWTPK7X4FIUO53V2FONNQYZEKGGBO5PXBA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:34.492220262 (209h32m34.363303307s) [noise=true , address=145.79.227.28:4725 , piece_id=ZDTVQMPD4EUWKRQQVP7PNAQ634L4MQBYJIAFPFSKLQUOTFZHMFPA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:35.590754704 (209h32m35.461837739s) [noise=true , address=145.79.227.46:4603 , piece_id=JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:35.590780002 (209h32m35.461863047s) [seg_pos={0 0}, upload=183567]: Segment reached optimal threshold of 13 pieces. Dec 14 09:01:35.591192537 (209h32m35.462275572s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: read tcp 188.166.225.170:47850->145.79.227.49:4339: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591202196 (209h32m35.462285231s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591227634 (209h32m35.462310669s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: rpc: dial tcp 145.79.227.39:4014: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591241200 (209h32m35.462324245s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591644728 (209h32m35.462727773s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:45550->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591655028 (209h32m35.462738073s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591681468 (209h32m35.462764503s) [seg_pos={0 0}, upload=183567]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 14 09:01:35.591768234 (209h32m35.462851279s) [upload=183567]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 14 09:01:35.591774916 (209h32m35.462857961s) [upload=183567]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 14 09:01:36.093868656 (209h32m35.964951701s) [upload=183567]: Done uploading segments... Dec 14 09:01:36.093895967 (209h32m35.964979012s) [upload=183567]: Upload finished. err: <nil> Duration: 8.470210541s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...XX.......................................................] duration: 7.301762623s trace: https://jaeger.dev.storj.io/trace/b3a498e2503e131 FAIL (2025-12-14 09:45:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ) context deadline exceeded; piecestore: (Node ID: 1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX, Piece ID: JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA) context deadline exceeded ======LOGS====== oad=183567]: Starting upload Dec 14 09:01:28.382436168 (209h32m28.253519203s) [upload=183567]: Done starting upload Dec 14 09:01:28.382504638 (209h32m28.253587683s) [upload=183567]: Uploading segments... Dec 14 09:01:28.383014028 (209h32m28.254097073s) [upload=183567]: Got next segment. Inline: false Dec 14 09:01:28.383065245 (209h32m28.254148281s) [upload=183567]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 09:01:28.383070747 (209h32m28.254153782s) [upload=183567]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 09:01:29.117351421 (209h32m28.988434477s) [seg_pos={0 0}, upload=183567]: Begin upload segment... Dec 14 09:01:29.117412969 (209h32m28.988496004s) [seg_pos={0 0}, upload=183567]: Done begin upload segment. Dec 14 09:01:29.117442345 (209h32m28.98852538s) [noise=true , address=145.79.227.26:4655 , piece_id=6YPJCS6WMI5AYEEESQSWTPK7X4FIUO53V2FONNQYZEKGGBO5PXBA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118002081 (209h32m28.989085116s) [noise=true , address=145.79.227.28:4725 , piece_id=ZDTVQMPD4EUWKRQQVP7PNAQ634L4MQBYJIAFPFSKLQUOTFZHMFPA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118088185 (209h32m28.98917122s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118501743 (209h32m28.989584788s) [noise=true , address=77.90.31.21:4878 , piece_id=HF6KZLDP62HJG7MWOAF4TTPXLI7C6CAUX5O4QWT5MP7ZWNZUWDDQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119009108 (209h32m28.990092143s) [noise=true , address=145.79.227.26:4653 , piece_id=RWUX47DDMWRTEXWKCDN6BF2PGSL6BZAXWQIX5DT547AKMWLNDWYQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119070265 (209h32m28.99015331s) [noise=true , address=145.79.227.47:4265 , piece_id=JLGBSCGU5PK7AAYNRKSJRLMQSKCG4SRM4WSVRALCW7GCA56LRFYQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119374144 (209h32m28.990457179s) [noise=true , address=77.90.31.14:4662 , piece_id=TQLMWNW5JAB62NZRPIMTM33N7GSAHZGCU3V7H32LZTSHE2SWMCIQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119462752 (209h32m28.990545797s) [noise=true , address=145.79.227.46:4601 , piece_id=NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119754849 (209h32m28.990837894s) [noise=true , address=145.79.227.41:4087 , piece_id=66RGE7N2TJ7PSWIBB2UDPIKNZ2ND4BZXFLEXDZWVMVMAYL5PSBGA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119815303 (209h32m28.990898338s) [noise=true , address=145.79.227.49:4337 , piece_id=HUJLCBY3PQDUJ4Y52LESV6CVZ5DMMCV4MO346NTVY3QCSG6NSZCQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120045883 (209h32m28.991128918s) [noise=true , address=145.79.227.31:4799 , piece_id=L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120254059 (209h32m28.991337094s) [noise=true , address=145.79.227.23:4546 , piece_id=3Q24KFURFMF5POQJ2MNRGWX32E6IKI6XW5XHSLTQ7CCXTEUTGW6A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120325726 (209h32m28.991408871s) [noise=true , address=145.79.227.42:4123 , piece_id=53K6JA7I4GAUEL3ROO65MKFXN7ZSNMOH22UGAXIBRW4WUKN7555A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120653820 (209h32m28.991736865s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.121122984 (209h32m28.992206029s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.121668723 (209h32m28.992751768s) [noise=true , address=145.79.227.46:4603 , piece_id=JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:30.977492703 (209h32m30.848575748s) [upload=183567]: Next returned nil segment Dec 14 09:01:30.977525565 (209h32m30.84860862s) [upload=183567]: Waiting for error group managing segments... Dec 14 09:01:31.270408579 (209h32m31.141491624s) [noise=true , address=145.79.227.42:4123 , piece_id=53K6JA7I4GAUEL3ROO65MKFXN7ZSNMOH22UGAXIBRW4WUKN7555A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.315656118 (209h32m31.186739163s) [noise=true , address=77.90.31.14:4662 , piece_id=TQLMWNW5JAB62NZRPIMTM33N7GSAHZGCU3V7H32LZTSHE2SWMCIQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.342981501 (209h32m31.214064536s) [noise=true , address=77.90.31.21:4878 , piece_id=HF6KZLDP62HJG7MWOAF4TTPXLI7C6CAUX5O4QWT5MP7ZWNZUWDDQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.739040250 (209h32m31.610123295s) [noise=true , address=145.79.227.23:4546 , piece_id=3Q24KFURFMF5POQJ2MNRGWX32E6IKI6XW5XHSLTQ7CCXTEUTGW6A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.800414969 (209h32m31.671498004s) [noise=true , address=145.79.227.47:4265 , piece_id=JLGBSCGU5PK7AAYNRKSJRLMQSKCG4SRM4WSVRALCW7GCA56LRFYQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:32.628091848 (209h32m32.499174883s) [noise=true , address=145.79.227.26:4653 , piece_id=RWUX47DDMWRTEXWKCDN6BF2PGSL6BZAXWQIX5DT547AKMWLNDWYQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:32.629981676 (209h32m32.501064721s) [noise=true , address=145.79.227.49:4337 , piece_id=HUJLCBY3PQDUJ4Y52LESV6CVZ5DMMCV4MO346NTVY3QCSG6NSZCQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.054036275 (209h32m32.92511931s) [noise=true , address=145.79.227.41:4087 , piece_id=66RGE7N2TJ7PSWIBB2UDPIKNZ2ND4BZXFLEXDZWVMVMAYL5PSBGA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.071010907 (209h32m32.942093952s) [noise=true , address=145.79.227.31:4799 , piece_id=L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.120830402 (209h32m32.991913438s) [noise=true , address=145.79.227.46:4601 , piece_id=NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:34.312983241 (209h32m34.184066276s) [noise=true , address=145.79.227.26:4655 , piece_id=6YPJCS6WMI5AYEEESQSWTPK7X4FIUO53V2FONNQYZEKGGBO5PXBA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:34.492220262 (209h32m34.363303307s) [noise=true , address=145.79.227.28:4725 , piece_id=ZDTVQMPD4EUWKRQQVP7PNAQ634L4MQBYJIAFPFSKLQUOTFZHMFPA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:35.590754704 (209h32m35.461837739s) [noise=true , address=145.79.227.46:4603 , piece_id=JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:35.590780002 (209h32m35.461863047s) [seg_pos={0 0}, upload=183567]: Segment reached optimal threshold of 13 pieces. Dec 14 09:01:35.591192537 (209h32m35.462275572s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: read tcp 188.166.225.170:47850->145.79.227.49:4339: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591202196 (209h32m35.462285231s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591227634 (209h32m35.462310669s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: rpc: dial tcp 145.79.227.39:4014: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591241200 (209h32m35.462324245s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591644728 (209h32m35.462727773s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:45550->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591655028 (209h32m35.462738073s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591681468 (209h32m35.462764503s) [seg_pos={0 0}, upload=183567]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 14 09:01:35.591768234 (209h32m35.462851279s) [upload=183567]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 14 09:01:35.591774916 (209h32m35.462857961s) [upload=183567]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 14 09:01:36.093868656 (209h32m35.964951701s) [upload=183567]: Done uploading segments... Dec 14 09:01:36.093895967 (209h32m35.964979012s) [upload=183567]: Upload finished. err: <nil> Duration: 7.301795304s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....XX......................................................] duration: 6.66362775s trace: https://jaeger.dev.storj.io/trace/b3a498e2503e131 FAIL (2025-12-14 09:45:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ) context deadline exceeded; piecestore: (Node ID: 1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX, Piece ID: JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA) context deadline exceeded ======LOGS====== oad=183567]: Starting upload Dec 14 09:01:28.382436168 (209h32m28.253519203s) [upload=183567]: Done starting upload Dec 14 09:01:28.382504638 (209h32m28.253587683s) [upload=183567]: Uploading segments... Dec 14 09:01:28.383014028 (209h32m28.254097073s) [upload=183567]: Got next segment. Inline: false Dec 14 09:01:28.383065245 (209h32m28.254148281s) [upload=183567]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 09:01:28.383070747 (209h32m28.254153782s) [upload=183567]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 09:01:29.117351421 (209h32m28.988434477s) [seg_pos={0 0}, upload=183567]: Begin upload segment... Dec 14 09:01:29.117412969 (209h32m28.988496004s) [seg_pos={0 0}, upload=183567]: Done begin upload segment. Dec 14 09:01:29.117442345 (209h32m28.98852538s) [noise=true , address=145.79.227.26:4655 , piece_id=6YPJCS6WMI5AYEEESQSWTPK7X4FIUO53V2FONNQYZEKGGBO5PXBA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118002081 (209h32m28.989085116s) [noise=true , address=145.79.227.28:4725 , piece_id=ZDTVQMPD4EUWKRQQVP7PNAQ634L4MQBYJIAFPFSKLQUOTFZHMFPA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118088185 (209h32m28.98917122s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118501743 (209h32m28.989584788s) [noise=true , address=77.90.31.21:4878 , piece_id=HF6KZLDP62HJG7MWOAF4TTPXLI7C6CAUX5O4QWT5MP7ZWNZUWDDQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119009108 (209h32m28.990092143s) [noise=true , address=145.79.227.26:4653 , piece_id=RWUX47DDMWRTEXWKCDN6BF2PGSL6BZAXWQIX5DT547AKMWLNDWYQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119070265 (209h32m28.99015331s) [noise=true , address=145.79.227.47:4265 , piece_id=JLGBSCGU5PK7AAYNRKSJRLMQSKCG4SRM4WSVRALCW7GCA56LRFYQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119374144 (209h32m28.990457179s) [noise=true , address=77.90.31.14:4662 , piece_id=TQLMWNW5JAB62NZRPIMTM33N7GSAHZGCU3V7H32LZTSHE2SWMCIQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119462752 (209h32m28.990545797s) [noise=true , address=145.79.227.46:4601 , piece_id=NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119754849 (209h32m28.990837894s) [noise=true , address=145.79.227.41:4087 , piece_id=66RGE7N2TJ7PSWIBB2UDPIKNZ2ND4BZXFLEXDZWVMVMAYL5PSBGA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119815303 (209h32m28.990898338s) [noise=true , address=145.79.227.49:4337 , piece_id=HUJLCBY3PQDUJ4Y52LESV6CVZ5DMMCV4MO346NTVY3QCSG6NSZCQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120045883 (209h32m28.991128918s) [noise=true , address=145.79.227.31:4799 , piece_id=L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120254059 (209h32m28.991337094s) [noise=true , address=145.79.227.23:4546 , piece_id=3Q24KFURFMF5POQJ2MNRGWX32E6IKI6XW5XHSLTQ7CCXTEUTGW6A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120325726 (209h32m28.991408871s) [noise=true , address=145.79.227.42:4123 , piece_id=53K6JA7I4GAUEL3ROO65MKFXN7ZSNMOH22UGAXIBRW4WUKN7555A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120653820 (209h32m28.991736865s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.121122984 (209h32m28.992206029s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.121668723 (209h32m28.992751768s) [noise=true , address=145.79.227.46:4603 , piece_id=JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:30.977492703 (209h32m30.848575748s) [upload=183567]: Next returned nil segment Dec 14 09:01:30.977525565 (209h32m30.84860862s) [upload=183567]: Waiting for error group managing segments... Dec 14 09:01:31.270408579 (209h32m31.141491624s) [noise=true , address=145.79.227.42:4123 , piece_id=53K6JA7I4GAUEL3ROO65MKFXN7ZSNMOH22UGAXIBRW4WUKN7555A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.315656118 (209h32m31.186739163s) [noise=true , address=77.90.31.14:4662 , piece_id=TQLMWNW5JAB62NZRPIMTM33N7GSAHZGCU3V7H32LZTSHE2SWMCIQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.342981501 (209h32m31.214064536s) [noise=true , address=77.90.31.21:4878 , piece_id=HF6KZLDP62HJG7MWOAF4TTPXLI7C6CAUX5O4QWT5MP7ZWNZUWDDQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.739040250 (209h32m31.610123295s) [noise=true , address=145.79.227.23:4546 , piece_id=3Q24KFURFMF5POQJ2MNRGWX32E6IKI6XW5XHSLTQ7CCXTEUTGW6A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.800414969 (209h32m31.671498004s) [noise=true , address=145.79.227.47:4265 , piece_id=JLGBSCGU5PK7AAYNRKSJRLMQSKCG4SRM4WSVRALCW7GCA56LRFYQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:32.628091848 (209h32m32.499174883s) [noise=true , address=145.79.227.26:4653 , piece_id=RWUX47DDMWRTEXWKCDN6BF2PGSL6BZAXWQIX5DT547AKMWLNDWYQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:32.629981676 (209h32m32.501064721s) [noise=true , address=145.79.227.49:4337 , piece_id=HUJLCBY3PQDUJ4Y52LESV6CVZ5DMMCV4MO346NTVY3QCSG6NSZCQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.054036275 (209h32m32.92511931s) [noise=true , address=145.79.227.41:4087 , piece_id=66RGE7N2TJ7PSWIBB2UDPIKNZ2ND4BZXFLEXDZWVMVMAYL5PSBGA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.071010907 (209h32m32.942093952s) [noise=true , address=145.79.227.31:4799 , piece_id=L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.120830402 (209h32m32.991913438s) [noise=true , address=145.79.227.46:4601 , piece_id=NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:34.312983241 (209h32m34.184066276s) [noise=true , address=145.79.227.26:4655 , piece_id=6YPJCS6WMI5AYEEESQSWTPK7X4FIUO53V2FONNQYZEKGGBO5PXBA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:34.492220262 (209h32m34.363303307s) [noise=true , address=145.79.227.28:4725 , piece_id=ZDTVQMPD4EUWKRQQVP7PNAQ634L4MQBYJIAFPFSKLQUOTFZHMFPA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:35.590754704 (209h32m35.461837739s) [noise=true , address=145.79.227.46:4603 , piece_id=JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:35.590780002 (209h32m35.461863047s) [seg_pos={0 0}, upload=183567]: Segment reached optimal threshold of 13 pieces. Dec 14 09:01:35.591192537 (209h32m35.462275572s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: read tcp 188.166.225.170:47850->145.79.227.49:4339: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591202196 (209h32m35.462285231s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591227634 (209h32m35.462310669s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: rpc: dial tcp 145.79.227.39:4014: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591241200 (209h32m35.462324245s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591644728 (209h32m35.462727773s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:45550->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591655028 (209h32m35.462738073s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591681468 (209h32m35.462764503s) [seg_pos={0 0}, upload=183567]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 14 09:01:35.591768234 (209h32m35.462851279s) [upload=183567]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 14 09:01:35.591774916 (209h32m35.462857961s) [upload=183567]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 14 09:01:36.093868656 (209h32m35.964951701s) [upload=183567]: Done uploading segments... Dec 14 09:01:36.093895967 (209h32m35.964979012s) [upload=183567]: Upload finished. err: <nil> Duration: 6.663660293s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....XX.....................................................] duration: 6.263814855s trace: https://jaeger.dev.storj.io/trace/b3a498e2503e131 FAIL (2025-12-14 09:44:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ) context deadline exceeded; piecestore: (Node ID: 1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX, Piece ID: JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA) context deadline exceeded ======LOGS====== oad=183567]: Starting upload Dec 14 09:01:28.382436168 (209h32m28.253519203s) [upload=183567]: Done starting upload Dec 14 09:01:28.382504638 (209h32m28.253587683s) [upload=183567]: Uploading segments... Dec 14 09:01:28.383014028 (209h32m28.254097073s) [upload=183567]: Got next segment. Inline: false Dec 14 09:01:28.383065245 (209h32m28.254148281s) [upload=183567]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 09:01:28.383070747 (209h32m28.254153782s) [upload=183567]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 09:01:29.117351421 (209h32m28.988434477s) [seg_pos={0 0}, upload=183567]: Begin upload segment... Dec 14 09:01:29.117412969 (209h32m28.988496004s) [seg_pos={0 0}, upload=183567]: Done begin upload segment. Dec 14 09:01:29.117442345 (209h32m28.98852538s) [noise=true , address=145.79.227.26:4655 , piece_id=6YPJCS6WMI5AYEEESQSWTPK7X4FIUO53V2FONNQYZEKGGBO5PXBA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118002081 (209h32m28.989085116s) [noise=true , address=145.79.227.28:4725 , piece_id=ZDTVQMPD4EUWKRQQVP7PNAQ634L4MQBYJIAFPFSKLQUOTFZHMFPA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118088185 (209h32m28.98917122s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118501743 (209h32m28.989584788s) [noise=true , address=77.90.31.21:4878 , piece_id=HF6KZLDP62HJG7MWOAF4TTPXLI7C6CAUX5O4QWT5MP7ZWNZUWDDQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119009108 (209h32m28.990092143s) [noise=true , address=145.79.227.26:4653 , piece_id=RWUX47DDMWRTEXWKCDN6BF2PGSL6BZAXWQIX5DT547AKMWLNDWYQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119070265 (209h32m28.99015331s) [noise=true , address=145.79.227.47:4265 , piece_id=JLGBSCGU5PK7AAYNRKSJRLMQSKCG4SRM4WSVRALCW7GCA56LRFYQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119374144 (209h32m28.990457179s) [noise=true , address=77.90.31.14:4662 , piece_id=TQLMWNW5JAB62NZRPIMTM33N7GSAHZGCU3V7H32LZTSHE2SWMCIQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119462752 (209h32m28.990545797s) [noise=true , address=145.79.227.46:4601 , piece_id=NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119754849 (209h32m28.990837894s) [noise=true , address=145.79.227.41:4087 , piece_id=66RGE7N2TJ7PSWIBB2UDPIKNZ2ND4BZXFLEXDZWVMVMAYL5PSBGA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119815303 (209h32m28.990898338s) [noise=true , address=145.79.227.49:4337 , piece_id=HUJLCBY3PQDUJ4Y52LESV6CVZ5DMMCV4MO346NTVY3QCSG6NSZCQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120045883 (209h32m28.991128918s) [noise=true , address=145.79.227.31:4799 , piece_id=L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120254059 (209h32m28.991337094s) [noise=true , address=145.79.227.23:4546 , piece_id=3Q24KFURFMF5POQJ2MNRGWX32E6IKI6XW5XHSLTQ7CCXTEUTGW6A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120325726 (209h32m28.991408871s) [noise=true , address=145.79.227.42:4123 , piece_id=53K6JA7I4GAUEL3ROO65MKFXN7ZSNMOH22UGAXIBRW4WUKN7555A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120653820 (209h32m28.991736865s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.121122984 (209h32m28.992206029s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.121668723 (209h32m28.992751768s) [noise=true , address=145.79.227.46:4603 , piece_id=JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:30.977492703 (209h32m30.848575748s) [upload=183567]: Next returned nil segment Dec 14 09:01:30.977525565 (209h32m30.84860862s) [upload=183567]: Waiting for error group managing segments... Dec 14 09:01:31.270408579 (209h32m31.141491624s) [noise=true , address=145.79.227.42:4123 , piece_id=53K6JA7I4GAUEL3ROO65MKFXN7ZSNMOH22UGAXIBRW4WUKN7555A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.315656118 (209h32m31.186739163s) [noise=true , address=77.90.31.14:4662 , piece_id=TQLMWNW5JAB62NZRPIMTM33N7GSAHZGCU3V7H32LZTSHE2SWMCIQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.342981501 (209h32m31.214064536s) [noise=true , address=77.90.31.21:4878 , piece_id=HF6KZLDP62HJG7MWOAF4TTPXLI7C6CAUX5O4QWT5MP7ZWNZUWDDQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.739040250 (209h32m31.610123295s) [noise=true , address=145.79.227.23:4546 , piece_id=3Q24KFURFMF5POQJ2MNRGWX32E6IKI6XW5XHSLTQ7CCXTEUTGW6A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.800414969 (209h32m31.671498004s) [noise=true , address=145.79.227.47:4265 , piece_id=JLGBSCGU5PK7AAYNRKSJRLMQSKCG4SRM4WSVRALCW7GCA56LRFYQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:32.628091848 (209h32m32.499174883s) [noise=true , address=145.79.227.26:4653 , piece_id=RWUX47DDMWRTEXWKCDN6BF2PGSL6BZAXWQIX5DT547AKMWLNDWYQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:32.629981676 (209h32m32.501064721s) [noise=true , address=145.79.227.49:4337 , piece_id=HUJLCBY3PQDUJ4Y52LESV6CVZ5DMMCV4MO346NTVY3QCSG6NSZCQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.054036275 (209h32m32.92511931s) [noise=true , address=145.79.227.41:4087 , piece_id=66RGE7N2TJ7PSWIBB2UDPIKNZ2ND4BZXFLEXDZWVMVMAYL5PSBGA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.071010907 (209h32m32.942093952s) [noise=true , address=145.79.227.31:4799 , piece_id=L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.120830402 (209h32m32.991913438s) [noise=true , address=145.79.227.46:4601 , piece_id=NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:34.312983241 (209h32m34.184066276s) [noise=true , address=145.79.227.26:4655 , piece_id=6YPJCS6WMI5AYEEESQSWTPK7X4FIUO53V2FONNQYZEKGGBO5PXBA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:34.492220262 (209h32m34.363303307s) [noise=true , address=145.79.227.28:4725 , piece_id=ZDTVQMPD4EUWKRQQVP7PNAQ634L4MQBYJIAFPFSKLQUOTFZHMFPA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:35.590754704 (209h32m35.461837739s) [noise=true , address=145.79.227.46:4603 , piece_id=JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:35.590780002 (209h32m35.461863047s) [seg_pos={0 0}, upload=183567]: Segment reached optimal threshold of 13 pieces. Dec 14 09:01:35.591192537 (209h32m35.462275572s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: read tcp 188.166.225.170:47850->145.79.227.49:4339: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591202196 (209h32m35.462285231s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591227634 (209h32m35.462310669s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: rpc: dial tcp 145.79.227.39:4014: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591241200 (209h32m35.462324245s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591644728 (209h32m35.462727773s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:45550->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591655028 (209h32m35.462738073s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591681468 (209h32m35.462764503s) [seg_pos={0 0}, upload=183567]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 14 09:01:35.591768234 (209h32m35.462851279s) [upload=183567]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 14 09:01:35.591774916 (209h32m35.462857961s) [upload=183567]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 14 09:01:36.093868656 (209h32m35.964951701s) [upload=183567]: Done uploading segments... Dec 14 09:01:36.093895967 (209h32m35.964979012s) [upload=183567]: Upload finished. err: <nil> Duration: 6.263842948s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.892840756s trace: https://jaeger.dev.storj.io/trace/9336cdef9668ab3 FAIL (2025-12-14 09:43:32 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.892867425s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......XX....................................................] duration: 7.128590129s trace: https://jaeger.dev.storj.io/trace/b3a498e2503e131 FAIL (2025-12-14 09:42:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ) context deadline exceeded; piecestore: (Node ID: 1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX, Piece ID: JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA) context deadline exceeded ======LOGS====== oad=183567]: Starting upload Dec 14 09:01:28.382436168 (209h32m28.253519203s) [upload=183567]: Done starting upload Dec 14 09:01:28.382504638 (209h32m28.253587683s) [upload=183567]: Uploading segments... Dec 14 09:01:28.383014028 (209h32m28.254097073s) [upload=183567]: Got next segment. Inline: false Dec 14 09:01:28.383065245 (209h32m28.254148281s) [upload=183567]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 09:01:28.383070747 (209h32m28.254153782s) [upload=183567]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 09:01:29.117351421 (209h32m28.988434477s) [seg_pos={0 0}, upload=183567]: Begin upload segment... Dec 14 09:01:29.117412969 (209h32m28.988496004s) [seg_pos={0 0}, upload=183567]: Done begin upload segment. Dec 14 09:01:29.117442345 (209h32m28.98852538s) [noise=true , address=145.79.227.26:4655 , piece_id=6YPJCS6WMI5AYEEESQSWTPK7X4FIUO53V2FONNQYZEKGGBO5PXBA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118002081 (209h32m28.989085116s) [noise=true , address=145.79.227.28:4725 , piece_id=ZDTVQMPD4EUWKRQQVP7PNAQ634L4MQBYJIAFPFSKLQUOTFZHMFPA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118088185 (209h32m28.98917122s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118501743 (209h32m28.989584788s) [noise=true , address=77.90.31.21:4878 , piece_id=HF6KZLDP62HJG7MWOAF4TTPXLI7C6CAUX5O4QWT5MP7ZWNZUWDDQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119009108 (209h32m28.990092143s) [noise=true , address=145.79.227.26:4653 , piece_id=RWUX47DDMWRTEXWKCDN6BF2PGSL6BZAXWQIX5DT547AKMWLNDWYQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119070265 (209h32m28.99015331s) [noise=true , address=145.79.227.47:4265 , piece_id=JLGBSCGU5PK7AAYNRKSJRLMQSKCG4SRM4WSVRALCW7GCA56LRFYQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119374144 (209h32m28.990457179s) [noise=true , address=77.90.31.14:4662 , piece_id=TQLMWNW5JAB62NZRPIMTM33N7GSAHZGCU3V7H32LZTSHE2SWMCIQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119462752 (209h32m28.990545797s) [noise=true , address=145.79.227.46:4601 , piece_id=NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119754849 (209h32m28.990837894s) [noise=true , address=145.79.227.41:4087 , piece_id=66RGE7N2TJ7PSWIBB2UDPIKNZ2ND4BZXFLEXDZWVMVMAYL5PSBGA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119815303 (209h32m28.990898338s) [noise=true , address=145.79.227.49:4337 , piece_id=HUJLCBY3PQDUJ4Y52LESV6CVZ5DMMCV4MO346NTVY3QCSG6NSZCQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120045883 (209h32m28.991128918s) [noise=true , address=145.79.227.31:4799 , piece_id=L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120254059 (209h32m28.991337094s) [noise=true , address=145.79.227.23:4546 , piece_id=3Q24KFURFMF5POQJ2MNRGWX32E6IKI6XW5XHSLTQ7CCXTEUTGW6A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120325726 (209h32m28.991408871s) [noise=true , address=145.79.227.42:4123 , piece_id=53K6JA7I4GAUEL3ROO65MKFXN7ZSNMOH22UGAXIBRW4WUKN7555A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120653820 (209h32m28.991736865s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.121122984 (209h32m28.992206029s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.121668723 (209h32m28.992751768s) [noise=true , address=145.79.227.46:4603 , piece_id=JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:30.977492703 (209h32m30.848575748s) [upload=183567]: Next returned nil segment Dec 14 09:01:30.977525565 (209h32m30.84860862s) [upload=183567]: Waiting for error group managing segments... Dec 14 09:01:31.270408579 (209h32m31.141491624s) [noise=true , address=145.79.227.42:4123 , piece_id=53K6JA7I4GAUEL3ROO65MKFXN7ZSNMOH22UGAXIBRW4WUKN7555A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.315656118 (209h32m31.186739163s) [noise=true , address=77.90.31.14:4662 , piece_id=TQLMWNW5JAB62NZRPIMTM33N7GSAHZGCU3V7H32LZTSHE2SWMCIQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.342981501 (209h32m31.214064536s) [noise=true , address=77.90.31.21:4878 , piece_id=HF6KZLDP62HJG7MWOAF4TTPXLI7C6CAUX5O4QWT5MP7ZWNZUWDDQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.739040250 (209h32m31.610123295s) [noise=true , address=145.79.227.23:4546 , piece_id=3Q24KFURFMF5POQJ2MNRGWX32E6IKI6XW5XHSLTQ7CCXTEUTGW6A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.800414969 (209h32m31.671498004s) [noise=true , address=145.79.227.47:4265 , piece_id=JLGBSCGU5PK7AAYNRKSJRLMQSKCG4SRM4WSVRALCW7GCA56LRFYQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:32.628091848 (209h32m32.499174883s) [noise=true , address=145.79.227.26:4653 , piece_id=RWUX47DDMWRTEXWKCDN6BF2PGSL6BZAXWQIX5DT547AKMWLNDWYQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:32.629981676 (209h32m32.501064721s) [noise=true , address=145.79.227.49:4337 , piece_id=HUJLCBY3PQDUJ4Y52LESV6CVZ5DMMCV4MO346NTVY3QCSG6NSZCQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.054036275 (209h32m32.92511931s) [noise=true , address=145.79.227.41:4087 , piece_id=66RGE7N2TJ7PSWIBB2UDPIKNZ2ND4BZXFLEXDZWVMVMAYL5PSBGA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.071010907 (209h32m32.942093952s) [noise=true , address=145.79.227.31:4799 , piece_id=L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.120830402 (209h32m32.991913438s) [noise=true , address=145.79.227.46:4601 , piece_id=NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:34.312983241 (209h32m34.184066276s) [noise=true , address=145.79.227.26:4655 , piece_id=6YPJCS6WMI5AYEEESQSWTPK7X4FIUO53V2FONNQYZEKGGBO5PXBA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:34.492220262 (209h32m34.363303307s) [noise=true , address=145.79.227.28:4725 , piece_id=ZDTVQMPD4EUWKRQQVP7PNAQ634L4MQBYJIAFPFSKLQUOTFZHMFPA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:35.590754704 (209h32m35.461837739s) [noise=true , address=145.79.227.46:4603 , piece_id=JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:35.590780002 (209h32m35.461863047s) [seg_pos={0 0}, upload=183567]: Segment reached optimal threshold of 13 pieces. Dec 14 09:01:35.591192537 (209h32m35.462275572s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: read tcp 188.166.225.170:47850->145.79.227.49:4339: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591202196 (209h32m35.462285231s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591227634 (209h32m35.462310669s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: rpc: dial tcp 145.79.227.39:4014: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591241200 (209h32m35.462324245s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591644728 (209h32m35.462727773s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:45550->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591655028 (209h32m35.462738073s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591681468 (209h32m35.462764503s) [seg_pos={0 0}, upload=183567]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 14 09:01:35.591768234 (209h32m35.462851279s) [upload=183567]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 14 09:01:35.591774916 (209h32m35.462857961s) [upload=183567]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 14 09:01:36.093868656 (209h32m35.964951701s) [upload=183567]: Done uploading segments... Dec 14 09:01:36.093895967 (209h32m35.964979012s) [upload=183567]: Upload finished. err: <nil> Duration: 7.128622499s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.985518939s trace: https://jaeger.dev.storj.io/trace/9336cdef9668ab3 FAIL (2025-12-14 09:42:38 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.985553776s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......XX...................................................] duration: 7.847187985s trace: https://jaeger.dev.storj.io/trace/b3a498e2503e131 FAIL (2025-12-14 09:41:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ) context deadline exceeded; piecestore: (Node ID: 1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX, Piece ID: JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA) context deadline exceeded ======LOGS====== oad=183567]: Starting upload Dec 14 09:01:28.382436168 (209h32m28.253519203s) [upload=183567]: Done starting upload Dec 14 09:01:28.382504638 (209h32m28.253587683s) [upload=183567]: Uploading segments... Dec 14 09:01:28.383014028 (209h32m28.254097073s) [upload=183567]: Got next segment. Inline: false Dec 14 09:01:28.383065245 (209h32m28.254148281s) [upload=183567]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 09:01:28.383070747 (209h32m28.254153782s) [upload=183567]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 09:01:29.117351421 (209h32m28.988434477s) [seg_pos={0 0}, upload=183567]: Begin upload segment... Dec 14 09:01:29.117412969 (209h32m28.988496004s) [seg_pos={0 0}, upload=183567]: Done begin upload segment. Dec 14 09:01:29.117442345 (209h32m28.98852538s) [noise=true , address=145.79.227.26:4655 , piece_id=6YPJCS6WMI5AYEEESQSWTPK7X4FIUO53V2FONNQYZEKGGBO5PXBA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118002081 (209h32m28.989085116s) [noise=true , address=145.79.227.28:4725 , piece_id=ZDTVQMPD4EUWKRQQVP7PNAQ634L4MQBYJIAFPFSKLQUOTFZHMFPA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118088185 (209h32m28.98917122s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118501743 (209h32m28.989584788s) [noise=true , address=77.90.31.21:4878 , piece_id=HF6KZLDP62HJG7MWOAF4TTPXLI7C6CAUX5O4QWT5MP7ZWNZUWDDQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119009108 (209h32m28.990092143s) [noise=true , address=145.79.227.26:4653 , piece_id=RWUX47DDMWRTEXWKCDN6BF2PGSL6BZAXWQIX5DT547AKMWLNDWYQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119070265 (209h32m28.99015331s) [noise=true , address=145.79.227.47:4265 , piece_id=JLGBSCGU5PK7AAYNRKSJRLMQSKCG4SRM4WSVRALCW7GCA56LRFYQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119374144 (209h32m28.990457179s) [noise=true , address=77.90.31.14:4662 , piece_id=TQLMWNW5JAB62NZRPIMTM33N7GSAHZGCU3V7H32LZTSHE2SWMCIQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119462752 (209h32m28.990545797s) [noise=true , address=145.79.227.46:4601 , piece_id=NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119754849 (209h32m28.990837894s) [noise=true , address=145.79.227.41:4087 , piece_id=66RGE7N2TJ7PSWIBB2UDPIKNZ2ND4BZXFLEXDZWVMVMAYL5PSBGA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119815303 (209h32m28.990898338s) [noise=true , address=145.79.227.49:4337 , piece_id=HUJLCBY3PQDUJ4Y52LESV6CVZ5DMMCV4MO346NTVY3QCSG6NSZCQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120045883 (209h32m28.991128918s) [noise=true , address=145.79.227.31:4799 , piece_id=L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120254059 (209h32m28.991337094s) [noise=true , address=145.79.227.23:4546 , piece_id=3Q24KFURFMF5POQJ2MNRGWX32E6IKI6XW5XHSLTQ7CCXTEUTGW6A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120325726 (209h32m28.991408871s) [noise=true , address=145.79.227.42:4123 , piece_id=53K6JA7I4GAUEL3ROO65MKFXN7ZSNMOH22UGAXIBRW4WUKN7555A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120653820 (209h32m28.991736865s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.121122984 (209h32m28.992206029s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.121668723 (209h32m28.992751768s) [noise=true , address=145.79.227.46:4603 , piece_id=JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:30.977492703 (209h32m30.848575748s) [upload=183567]: Next returned nil segment Dec 14 09:01:30.977525565 (209h32m30.84860862s) [upload=183567]: Waiting for error group managing segments... Dec 14 09:01:31.270408579 (209h32m31.141491624s) [noise=true , address=145.79.227.42:4123 , piece_id=53K6JA7I4GAUEL3ROO65MKFXN7ZSNMOH22UGAXIBRW4WUKN7555A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.315656118 (209h32m31.186739163s) [noise=true , address=77.90.31.14:4662 , piece_id=TQLMWNW5JAB62NZRPIMTM33N7GSAHZGCU3V7H32LZTSHE2SWMCIQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.342981501 (209h32m31.214064536s) [noise=true , address=77.90.31.21:4878 , piece_id=HF6KZLDP62HJG7MWOAF4TTPXLI7C6CAUX5O4QWT5MP7ZWNZUWDDQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.739040250 (209h32m31.610123295s) [noise=true , address=145.79.227.23:4546 , piece_id=3Q24KFURFMF5POQJ2MNRGWX32E6IKI6XW5XHSLTQ7CCXTEUTGW6A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.800414969 (209h32m31.671498004s) [noise=true , address=145.79.227.47:4265 , piece_id=JLGBSCGU5PK7AAYNRKSJRLMQSKCG4SRM4WSVRALCW7GCA56LRFYQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:32.628091848 (209h32m32.499174883s) [noise=true , address=145.79.227.26:4653 , piece_id=RWUX47DDMWRTEXWKCDN6BF2PGSL6BZAXWQIX5DT547AKMWLNDWYQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:32.629981676 (209h32m32.501064721s) [noise=true , address=145.79.227.49:4337 , piece_id=HUJLCBY3PQDUJ4Y52LESV6CVZ5DMMCV4MO346NTVY3QCSG6NSZCQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.054036275 (209h32m32.92511931s) [noise=true , address=145.79.227.41:4087 , piece_id=66RGE7N2TJ7PSWIBB2UDPIKNZ2ND4BZXFLEXDZWVMVMAYL5PSBGA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.071010907 (209h32m32.942093952s) [noise=true , address=145.79.227.31:4799 , piece_id=L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.120830402 (209h32m32.991913438s) [noise=true , address=145.79.227.46:4601 , piece_id=NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:34.312983241 (209h32m34.184066276s) [noise=true , address=145.79.227.26:4655 , piece_id=6YPJCS6WMI5AYEEESQSWTPK7X4FIUO53V2FONNQYZEKGGBO5PXBA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:34.492220262 (209h32m34.363303307s) [noise=true , address=145.79.227.28:4725 , piece_id=ZDTVQMPD4EUWKRQQVP7PNAQ634L4MQBYJIAFPFSKLQUOTFZHMFPA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:35.590754704 (209h32m35.461837739s) [noise=true , address=145.79.227.46:4603 , piece_id=JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:35.590780002 (209h32m35.461863047s) [seg_pos={0 0}, upload=183567]: Segment reached optimal threshold of 13 pieces. Dec 14 09:01:35.591192537 (209h32m35.462275572s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: read tcp 188.166.225.170:47850->145.79.227.49:4339: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591202196 (209h32m35.462285231s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591227634 (209h32m35.462310669s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: rpc: dial tcp 145.79.227.39:4014: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591241200 (209h32m35.462324245s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591644728 (209h32m35.462727773s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:45550->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591655028 (209h32m35.462738073s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591681468 (209h32m35.462764503s) [seg_pos={0 0}, upload=183567]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 14 09:01:35.591768234 (209h32m35.462851279s) [upload=183567]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 14 09:01:35.591774916 (209h32m35.462857961s) [upload=183567]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 14 09:01:36.093868656 (209h32m35.964951701s) [upload=183567]: Done uploading segments... Dec 14 09:01:36.093895967 (209h32m35.964979012s) [upload=183567]: Upload finished. err: <nil> Duration: 7.84721129s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.055498473s trace: https://jaeger.dev.storj.io/trace/9336cdef9668ab3 FAIL (2025-12-14 09:41:32 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.055529503s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........XX..................................................] duration: 7.927079323s trace: https://jaeger.dev.storj.io/trace/b3a498e2503e131 FAIL (2025-12-14 09:41:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ) context deadline exceeded; piecestore: (Node ID: 1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX, Piece ID: JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA) context deadline exceeded ======LOGS====== oad=183567]: Starting upload Dec 14 09:01:28.382436168 (209h32m28.253519203s) [upload=183567]: Done starting upload Dec 14 09:01:28.382504638 (209h32m28.253587683s) [upload=183567]: Uploading segments... Dec 14 09:01:28.383014028 (209h32m28.254097073s) [upload=183567]: Got next segment. Inline: false Dec 14 09:01:28.383065245 (209h32m28.254148281s) [upload=183567]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 09:01:28.383070747 (209h32m28.254153782s) [upload=183567]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 09:01:29.117351421 (209h32m28.988434477s) [seg_pos={0 0}, upload=183567]: Begin upload segment... Dec 14 09:01:29.117412969 (209h32m28.988496004s) [seg_pos={0 0}, upload=183567]: Done begin upload segment. Dec 14 09:01:29.117442345 (209h32m28.98852538s) [noise=true , address=145.79.227.26:4655 , piece_id=6YPJCS6WMI5AYEEESQSWTPK7X4FIUO53V2FONNQYZEKGGBO5PXBA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118002081 (209h32m28.989085116s) [noise=true , address=145.79.227.28:4725 , piece_id=ZDTVQMPD4EUWKRQQVP7PNAQ634L4MQBYJIAFPFSKLQUOTFZHMFPA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118088185 (209h32m28.98917122s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118501743 (209h32m28.989584788s) [noise=true , address=77.90.31.21:4878 , piece_id=HF6KZLDP62HJG7MWOAF4TTPXLI7C6CAUX5O4QWT5MP7ZWNZUWDDQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119009108 (209h32m28.990092143s) [noise=true , address=145.79.227.26:4653 , piece_id=RWUX47DDMWRTEXWKCDN6BF2PGSL6BZAXWQIX5DT547AKMWLNDWYQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119070265 (209h32m28.99015331s) [noise=true , address=145.79.227.47:4265 , piece_id=JLGBSCGU5PK7AAYNRKSJRLMQSKCG4SRM4WSVRALCW7GCA56LRFYQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119374144 (209h32m28.990457179s) [noise=true , address=77.90.31.14:4662 , piece_id=TQLMWNW5JAB62NZRPIMTM33N7GSAHZGCU3V7H32LZTSHE2SWMCIQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119462752 (209h32m28.990545797s) [noise=true , address=145.79.227.46:4601 , piece_id=NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119754849 (209h32m28.990837894s) [noise=true , address=145.79.227.41:4087 , piece_id=66RGE7N2TJ7PSWIBB2UDPIKNZ2ND4BZXFLEXDZWVMVMAYL5PSBGA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119815303 (209h32m28.990898338s) [noise=true , address=145.79.227.49:4337 , piece_id=HUJLCBY3PQDUJ4Y52LESV6CVZ5DMMCV4MO346NTVY3QCSG6NSZCQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120045883 (209h32m28.991128918s) [noise=true , address=145.79.227.31:4799 , piece_id=L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120254059 (209h32m28.991337094s) [noise=true , address=145.79.227.23:4546 , piece_id=3Q24KFURFMF5POQJ2MNRGWX32E6IKI6XW5XHSLTQ7CCXTEUTGW6A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120325726 (209h32m28.991408871s) [noise=true , address=145.79.227.42:4123 , piece_id=53K6JA7I4GAUEL3ROO65MKFXN7ZSNMOH22UGAXIBRW4WUKN7555A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120653820 (209h32m28.991736865s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.121122984 (209h32m28.992206029s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.121668723 (209h32m28.992751768s) [noise=true , address=145.79.227.46:4603 , piece_id=JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:30.977492703 (209h32m30.848575748s) [upload=183567]: Next returned nil segment Dec 14 09:01:30.977525565 (209h32m30.84860862s) [upload=183567]: Waiting for error group managing segments... Dec 14 09:01:31.270408579 (209h32m31.141491624s) [noise=true , address=145.79.227.42:4123 , piece_id=53K6JA7I4GAUEL3ROO65MKFXN7ZSNMOH22UGAXIBRW4WUKN7555A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.315656118 (209h32m31.186739163s) [noise=true , address=77.90.31.14:4662 , piece_id=TQLMWNW5JAB62NZRPIMTM33N7GSAHZGCU3V7H32LZTSHE2SWMCIQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.342981501 (209h32m31.214064536s) [noise=true , address=77.90.31.21:4878 , piece_id=HF6KZLDP62HJG7MWOAF4TTPXLI7C6CAUX5O4QWT5MP7ZWNZUWDDQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.739040250 (209h32m31.610123295s) [noise=true , address=145.79.227.23:4546 , piece_id=3Q24KFURFMF5POQJ2MNRGWX32E6IKI6XW5XHSLTQ7CCXTEUTGW6A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.800414969 (209h32m31.671498004s) [noise=true , address=145.79.227.47:4265 , piece_id=JLGBSCGU5PK7AAYNRKSJRLMQSKCG4SRM4WSVRALCW7GCA56LRFYQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:32.628091848 (209h32m32.499174883s) [noise=true , address=145.79.227.26:4653 , piece_id=RWUX47DDMWRTEXWKCDN6BF2PGSL6BZAXWQIX5DT547AKMWLNDWYQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:32.629981676 (209h32m32.501064721s) [noise=true , address=145.79.227.49:4337 , piece_id=HUJLCBY3PQDUJ4Y52LESV6CVZ5DMMCV4MO346NTVY3QCSG6NSZCQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.054036275 (209h32m32.92511931s) [noise=true , address=145.79.227.41:4087 , piece_id=66RGE7N2TJ7PSWIBB2UDPIKNZ2ND4BZXFLEXDZWVMVMAYL5PSBGA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.071010907 (209h32m32.942093952s) [noise=true , address=145.79.227.31:4799 , piece_id=L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.120830402 (209h32m32.991913438s) [noise=true , address=145.79.227.46:4601 , piece_id=NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:34.312983241 (209h32m34.184066276s) [noise=true , address=145.79.227.26:4655 , piece_id=6YPJCS6WMI5AYEEESQSWTPK7X4FIUO53V2FONNQYZEKGGBO5PXBA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:34.492220262 (209h32m34.363303307s) [noise=true , address=145.79.227.28:4725 , piece_id=ZDTVQMPD4EUWKRQQVP7PNAQ634L4MQBYJIAFPFSKLQUOTFZHMFPA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:35.590754704 (209h32m35.461837739s) [noise=true , address=145.79.227.46:4603 , piece_id=JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:35.590780002 (209h32m35.461863047s) [seg_pos={0 0}, upload=183567]: Segment reached optimal threshold of 13 pieces. Dec 14 09:01:35.591192537 (209h32m35.462275572s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: read tcp 188.166.225.170:47850->145.79.227.49:4339: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591202196 (209h32m35.462285231s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591227634 (209h32m35.462310669s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: rpc: dial tcp 145.79.227.39:4014: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591241200 (209h32m35.462324245s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591644728 (209h32m35.462727773s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:45550->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591655028 (209h32m35.462738073s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591681468 (209h32m35.462764503s) [seg_pos={0 0}, upload=183567]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 14 09:01:35.591768234 (209h32m35.462851279s) [upload=183567]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 14 09:01:35.591774916 (209h32m35.462857961s) [upload=183567]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 14 09:01:36.093868656 (209h32m35.964951701s) [upload=183567]: Done uploading segments... Dec 14 09:01:36.093895967 (209h32m35.964979012s) [upload=183567]: Upload finished. err: <nil> Duration: 7.927108408s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.810892086s trace: https://jaeger.dev.storj.io/trace/9336cdef9668ab3 FAIL (2025-12-14 09:40:42 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.810918066s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........XX.................................................] duration: 6.364581083s trace: https://jaeger.dev.storj.io/trace/b3a498e2503e131 FAIL (2025-12-14 09:40:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ) context deadline exceeded; piecestore: (Node ID: 1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX, Piece ID: JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA) context deadline exceeded ======LOGS====== oad=183567]: Starting upload Dec 14 09:01:28.382436168 (209h32m28.253519203s) [upload=183567]: Done starting upload Dec 14 09:01:28.382504638 (209h32m28.253587683s) [upload=183567]: Uploading segments... Dec 14 09:01:28.383014028 (209h32m28.254097073s) [upload=183567]: Got next segment. Inline: false Dec 14 09:01:28.383065245 (209h32m28.254148281s) [upload=183567]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 09:01:28.383070747 (209h32m28.254153782s) [upload=183567]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 09:01:29.117351421 (209h32m28.988434477s) [seg_pos={0 0}, upload=183567]: Begin upload segment... Dec 14 09:01:29.117412969 (209h32m28.988496004s) [seg_pos={0 0}, upload=183567]: Done begin upload segment. Dec 14 09:01:29.117442345 (209h32m28.98852538s) [noise=true , address=145.79.227.26:4655 , piece_id=6YPJCS6WMI5AYEEESQSWTPK7X4FIUO53V2FONNQYZEKGGBO5PXBA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118002081 (209h32m28.989085116s) [noise=true , address=145.79.227.28:4725 , piece_id=ZDTVQMPD4EUWKRQQVP7PNAQ634L4MQBYJIAFPFSKLQUOTFZHMFPA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118088185 (209h32m28.98917122s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118501743 (209h32m28.989584788s) [noise=true , address=77.90.31.21:4878 , piece_id=HF6KZLDP62HJG7MWOAF4TTPXLI7C6CAUX5O4QWT5MP7ZWNZUWDDQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119009108 (209h32m28.990092143s) [noise=true , address=145.79.227.26:4653 , piece_id=RWUX47DDMWRTEXWKCDN6BF2PGSL6BZAXWQIX5DT547AKMWLNDWYQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119070265 (209h32m28.99015331s) [noise=true , address=145.79.227.47:4265 , piece_id=JLGBSCGU5PK7AAYNRKSJRLMQSKCG4SRM4WSVRALCW7GCA56LRFYQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119374144 (209h32m28.990457179s) [noise=true , address=77.90.31.14:4662 , piece_id=TQLMWNW5JAB62NZRPIMTM33N7GSAHZGCU3V7H32LZTSHE2SWMCIQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119462752 (209h32m28.990545797s) [noise=true , address=145.79.227.46:4601 , piece_id=NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119754849 (209h32m28.990837894s) [noise=true , address=145.79.227.41:4087 , piece_id=66RGE7N2TJ7PSWIBB2UDPIKNZ2ND4BZXFLEXDZWVMVMAYL5PSBGA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119815303 (209h32m28.990898338s) [noise=true , address=145.79.227.49:4337 , piece_id=HUJLCBY3PQDUJ4Y52LESV6CVZ5DMMCV4MO346NTVY3QCSG6NSZCQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120045883 (209h32m28.991128918s) [noise=true , address=145.79.227.31:4799 , piece_id=L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120254059 (209h32m28.991337094s) [noise=true , address=145.79.227.23:4546 , piece_id=3Q24KFURFMF5POQJ2MNRGWX32E6IKI6XW5XHSLTQ7CCXTEUTGW6A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120325726 (209h32m28.991408871s) [noise=true , address=145.79.227.42:4123 , piece_id=53K6JA7I4GAUEL3ROO65MKFXN7ZSNMOH22UGAXIBRW4WUKN7555A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120653820 (209h32m28.991736865s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.121122984 (209h32m28.992206029s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.121668723 (209h32m28.992751768s) [noise=true , address=145.79.227.46:4603 , piece_id=JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:30.977492703 (209h32m30.848575748s) [upload=183567]: Next returned nil segment Dec 14 09:01:30.977525565 (209h32m30.84860862s) [upload=183567]: Waiting for error group managing segments... Dec 14 09:01:31.270408579 (209h32m31.141491624s) [noise=true , address=145.79.227.42:4123 , piece_id=53K6JA7I4GAUEL3ROO65MKFXN7ZSNMOH22UGAXIBRW4WUKN7555A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.315656118 (209h32m31.186739163s) [noise=true , address=77.90.31.14:4662 , piece_id=TQLMWNW5JAB62NZRPIMTM33N7GSAHZGCU3V7H32LZTSHE2SWMCIQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.342981501 (209h32m31.214064536s) [noise=true , address=77.90.31.21:4878 , piece_id=HF6KZLDP62HJG7MWOAF4TTPXLI7C6CAUX5O4QWT5MP7ZWNZUWDDQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.739040250 (209h32m31.610123295s) [noise=true , address=145.79.227.23:4546 , piece_id=3Q24KFURFMF5POQJ2MNRGWX32E6IKI6XW5XHSLTQ7CCXTEUTGW6A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.800414969 (209h32m31.671498004s) [noise=true , address=145.79.227.47:4265 , piece_id=JLGBSCGU5PK7AAYNRKSJRLMQSKCG4SRM4WSVRALCW7GCA56LRFYQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:32.628091848 (209h32m32.499174883s) [noise=true , address=145.79.227.26:4653 , piece_id=RWUX47DDMWRTEXWKCDN6BF2PGSL6BZAXWQIX5DT547AKMWLNDWYQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:32.629981676 (209h32m32.501064721s) [noise=true , address=145.79.227.49:4337 , piece_id=HUJLCBY3PQDUJ4Y52LESV6CVZ5DMMCV4MO346NTVY3QCSG6NSZCQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.054036275 (209h32m32.92511931s) [noise=true , address=145.79.227.41:4087 , piece_id=66RGE7N2TJ7PSWIBB2UDPIKNZ2ND4BZXFLEXDZWVMVMAYL5PSBGA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.071010907 (209h32m32.942093952s) [noise=true , address=145.79.227.31:4799 , piece_id=L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.120830402 (209h32m32.991913438s) [noise=true , address=145.79.227.46:4601 , piece_id=NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:34.312983241 (209h32m34.184066276s) [noise=true , address=145.79.227.26:4655 , piece_id=6YPJCS6WMI5AYEEESQSWTPK7X4FIUO53V2FONNQYZEKGGBO5PXBA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:34.492220262 (209h32m34.363303307s) [noise=true , address=145.79.227.28:4725 , piece_id=ZDTVQMPD4EUWKRQQVP7PNAQ634L4MQBYJIAFPFSKLQUOTFZHMFPA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:35.590754704 (209h32m35.461837739s) [noise=true , address=145.79.227.46:4603 , piece_id=JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:35.590780002 (209h32m35.461863047s) [seg_pos={0 0}, upload=183567]: Segment reached optimal threshold of 13 pieces. Dec 14 09:01:35.591192537 (209h32m35.462275572s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: read tcp 188.166.225.170:47850->145.79.227.49:4339: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591202196 (209h32m35.462285231s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591227634 (209h32m35.462310669s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: rpc: dial tcp 145.79.227.39:4014: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591241200 (209h32m35.462324245s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591644728 (209h32m35.462727773s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:45550->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591655028 (209h32m35.462738073s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591681468 (209h32m35.462764503s) [seg_pos={0 0}, upload=183567]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 14 09:01:35.591768234 (209h32m35.462851279s) [upload=183567]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 14 09:01:35.591774916 (209h32m35.462857961s) [upload=183567]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 14 09:01:36.093868656 (209h32m35.964951701s) [upload=183567]: Done uploading segments... Dec 14 09:01:36.093895967 (209h32m35.964979012s) [upload=183567]: Upload finished. err: <nil> Duration: 6.364610087s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 7.338248036s trace: https://jaeger.dev.storj.io/trace/9336cdef9668ab3 FAIL (2025-12-14 09:40:16 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 7.338281531s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........XX................................................] duration: 6.321819305s trace: https://jaeger.dev.storj.io/trace/b3a498e2503e131 FAIL (2025-12-14 09:39:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ) context deadline exceeded; piecestore: (Node ID: 1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX, Piece ID: JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA) context deadline exceeded ======LOGS====== oad=183567]: Starting upload Dec 14 09:01:28.382436168 (209h32m28.253519203s) [upload=183567]: Done starting upload Dec 14 09:01:28.382504638 (209h32m28.253587683s) [upload=183567]: Uploading segments... Dec 14 09:01:28.383014028 (209h32m28.254097073s) [upload=183567]: Got next segment. Inline: false Dec 14 09:01:28.383065245 (209h32m28.254148281s) [upload=183567]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 09:01:28.383070747 (209h32m28.254153782s) [upload=183567]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 09:01:29.117351421 (209h32m28.988434477s) [seg_pos={0 0}, upload=183567]: Begin upload segment... Dec 14 09:01:29.117412969 (209h32m28.988496004s) [seg_pos={0 0}, upload=183567]: Done begin upload segment. Dec 14 09:01:29.117442345 (209h32m28.98852538s) [noise=true , address=145.79.227.26:4655 , piece_id=6YPJCS6WMI5AYEEESQSWTPK7X4FIUO53V2FONNQYZEKGGBO5PXBA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118002081 (209h32m28.989085116s) [noise=true , address=145.79.227.28:4725 , piece_id=ZDTVQMPD4EUWKRQQVP7PNAQ634L4MQBYJIAFPFSKLQUOTFZHMFPA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118088185 (209h32m28.98917122s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118501743 (209h32m28.989584788s) [noise=true , address=77.90.31.21:4878 , piece_id=HF6KZLDP62HJG7MWOAF4TTPXLI7C6CAUX5O4QWT5MP7ZWNZUWDDQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119009108 (209h32m28.990092143s) [noise=true , address=145.79.227.26:4653 , piece_id=RWUX47DDMWRTEXWKCDN6BF2PGSL6BZAXWQIX5DT547AKMWLNDWYQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119070265 (209h32m28.99015331s) [noise=true , address=145.79.227.47:4265 , piece_id=JLGBSCGU5PK7AAYNRKSJRLMQSKCG4SRM4WSVRALCW7GCA56LRFYQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119374144 (209h32m28.990457179s) [noise=true , address=77.90.31.14:4662 , piece_id=TQLMWNW5JAB62NZRPIMTM33N7GSAHZGCU3V7H32LZTSHE2SWMCIQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119462752 (209h32m28.990545797s) [noise=true , address=145.79.227.46:4601 , piece_id=NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119754849 (209h32m28.990837894s) [noise=true , address=145.79.227.41:4087 , piece_id=66RGE7N2TJ7PSWIBB2UDPIKNZ2ND4BZXFLEXDZWVMVMAYL5PSBGA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119815303 (209h32m28.990898338s) [noise=true , address=145.79.227.49:4337 , piece_id=HUJLCBY3PQDUJ4Y52LESV6CVZ5DMMCV4MO346NTVY3QCSG6NSZCQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120045883 (209h32m28.991128918s) [noise=true , address=145.79.227.31:4799 , piece_id=L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120254059 (209h32m28.991337094s) [noise=true , address=145.79.227.23:4546 , piece_id=3Q24KFURFMF5POQJ2MNRGWX32E6IKI6XW5XHSLTQ7CCXTEUTGW6A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120325726 (209h32m28.991408871s) [noise=true , address=145.79.227.42:4123 , piece_id=53K6JA7I4GAUEL3ROO65MKFXN7ZSNMOH22UGAXIBRW4WUKN7555A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120653820 (209h32m28.991736865s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.121122984 (209h32m28.992206029s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.121668723 (209h32m28.992751768s) [noise=true , address=145.79.227.46:4603 , piece_id=JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:30.977492703 (209h32m30.848575748s) [upload=183567]: Next returned nil segment Dec 14 09:01:30.977525565 (209h32m30.84860862s) [upload=183567]: Waiting for error group managing segments... Dec 14 09:01:31.270408579 (209h32m31.141491624s) [noise=true , address=145.79.227.42:4123 , piece_id=53K6JA7I4GAUEL3ROO65MKFXN7ZSNMOH22UGAXIBRW4WUKN7555A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.315656118 (209h32m31.186739163s) [noise=true , address=77.90.31.14:4662 , piece_id=TQLMWNW5JAB62NZRPIMTM33N7GSAHZGCU3V7H32LZTSHE2SWMCIQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.342981501 (209h32m31.214064536s) [noise=true , address=77.90.31.21:4878 , piece_id=HF6KZLDP62HJG7MWOAF4TTPXLI7C6CAUX5O4QWT5MP7ZWNZUWDDQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.739040250 (209h32m31.610123295s) [noise=true , address=145.79.227.23:4546 , piece_id=3Q24KFURFMF5POQJ2MNRGWX32E6IKI6XW5XHSLTQ7CCXTEUTGW6A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.800414969 (209h32m31.671498004s) [noise=true , address=145.79.227.47:4265 , piece_id=JLGBSCGU5PK7AAYNRKSJRLMQSKCG4SRM4WSVRALCW7GCA56LRFYQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:32.628091848 (209h32m32.499174883s) [noise=true , address=145.79.227.26:4653 , piece_id=RWUX47DDMWRTEXWKCDN6BF2PGSL6BZAXWQIX5DT547AKMWLNDWYQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:32.629981676 (209h32m32.501064721s) [noise=true , address=145.79.227.49:4337 , piece_id=HUJLCBY3PQDUJ4Y52LESV6CVZ5DMMCV4MO346NTVY3QCSG6NSZCQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.054036275 (209h32m32.92511931s) [noise=true , address=145.79.227.41:4087 , piece_id=66RGE7N2TJ7PSWIBB2UDPIKNZ2ND4BZXFLEXDZWVMVMAYL5PSBGA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.071010907 (209h32m32.942093952s) [noise=true , address=145.79.227.31:4799 , piece_id=L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.120830402 (209h32m32.991913438s) [noise=true , address=145.79.227.46:4601 , piece_id=NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:34.312983241 (209h32m34.184066276s) [noise=true , address=145.79.227.26:4655 , piece_id=6YPJCS6WMI5AYEEESQSWTPK7X4FIUO53V2FONNQYZEKGGBO5PXBA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:34.492220262 (209h32m34.363303307s) [noise=true , address=145.79.227.28:4725 , piece_id=ZDTVQMPD4EUWKRQQVP7PNAQ634L4MQBYJIAFPFSKLQUOTFZHMFPA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:35.590754704 (209h32m35.461837739s) [noise=true , address=145.79.227.46:4603 , piece_id=JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:35.590780002 (209h32m35.461863047s) [seg_pos={0 0}, upload=183567]: Segment reached optimal threshold of 13 pieces. Dec 14 09:01:35.591192537 (209h32m35.462275572s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: read tcp 188.166.225.170:47850->145.79.227.49:4339: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591202196 (209h32m35.462285231s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591227634 (209h32m35.462310669s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: rpc: dial tcp 145.79.227.39:4014: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591241200 (209h32m35.462324245s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591644728 (209h32m35.462727773s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:45550->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591655028 (209h32m35.462738073s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591681468 (209h32m35.462764503s) [seg_pos={0 0}, upload=183567]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 14 09:01:35.591768234 (209h32m35.462851279s) [upload=183567]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 14 09:01:35.591774916 (209h32m35.462857961s) [upload=183567]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 14 09:01:36.093868656 (209h32m35.964951701s) [upload=183567]: Done uploading segments... Dec 14 09:01:36.093895967 (209h32m35.964979012s) [upload=183567]: Upload finished. err: <nil> Duration: 6.321919165s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.286928929s trace: https://jaeger.dev.storj.io/trace/9336cdef9668ab3 FAIL (2025-12-14 09:39:22 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.286957473s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........XX...............................................] duration: 7.954395306s trace: https://jaeger.dev.storj.io/trace/b3a498e2503e131 FAIL (2025-12-14 09:39:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ) context deadline exceeded; piecestore: (Node ID: 1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX, Piece ID: JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA) context deadline exceeded ======LOGS====== oad=183567]: Starting upload Dec 14 09:01:28.382436168 (209h32m28.253519203s) [upload=183567]: Done starting upload Dec 14 09:01:28.382504638 (209h32m28.253587683s) [upload=183567]: Uploading segments... Dec 14 09:01:28.383014028 (209h32m28.254097073s) [upload=183567]: Got next segment. Inline: false Dec 14 09:01:28.383065245 (209h32m28.254148281s) [upload=183567]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 09:01:28.383070747 (209h32m28.254153782s) [upload=183567]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 09:01:29.117351421 (209h32m28.988434477s) [seg_pos={0 0}, upload=183567]: Begin upload segment... Dec 14 09:01:29.117412969 (209h32m28.988496004s) [seg_pos={0 0}, upload=183567]: Done begin upload segment. Dec 14 09:01:29.117442345 (209h32m28.98852538s) [noise=true , address=145.79.227.26:4655 , piece_id=6YPJCS6WMI5AYEEESQSWTPK7X4FIUO53V2FONNQYZEKGGBO5PXBA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118002081 (209h32m28.989085116s) [noise=true , address=145.79.227.28:4725 , piece_id=ZDTVQMPD4EUWKRQQVP7PNAQ634L4MQBYJIAFPFSKLQUOTFZHMFPA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118088185 (209h32m28.98917122s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118501743 (209h32m28.989584788s) [noise=true , address=77.90.31.21:4878 , piece_id=HF6KZLDP62HJG7MWOAF4TTPXLI7C6CAUX5O4QWT5MP7ZWNZUWDDQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119009108 (209h32m28.990092143s) [noise=true , address=145.79.227.26:4653 , piece_id=RWUX47DDMWRTEXWKCDN6BF2PGSL6BZAXWQIX5DT547AKMWLNDWYQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119070265 (209h32m28.99015331s) [noise=true , address=145.79.227.47:4265 , piece_id=JLGBSCGU5PK7AAYNRKSJRLMQSKCG4SRM4WSVRALCW7GCA56LRFYQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119374144 (209h32m28.990457179s) [noise=true , address=77.90.31.14:4662 , piece_id=TQLMWNW5JAB62NZRPIMTM33N7GSAHZGCU3V7H32LZTSHE2SWMCIQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119462752 (209h32m28.990545797s) [noise=true , address=145.79.227.46:4601 , piece_id=NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119754849 (209h32m28.990837894s) [noise=true , address=145.79.227.41:4087 , piece_id=66RGE7N2TJ7PSWIBB2UDPIKNZ2ND4BZXFLEXDZWVMVMAYL5PSBGA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119815303 (209h32m28.990898338s) [noise=true , address=145.79.227.49:4337 , piece_id=HUJLCBY3PQDUJ4Y52LESV6CVZ5DMMCV4MO346NTVY3QCSG6NSZCQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120045883 (209h32m28.991128918s) [noise=true , address=145.79.227.31:4799 , piece_id=L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120254059 (209h32m28.991337094s) [noise=true , address=145.79.227.23:4546 , piece_id=3Q24KFURFMF5POQJ2MNRGWX32E6IKI6XW5XHSLTQ7CCXTEUTGW6A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120325726 (209h32m28.991408871s) [noise=true , address=145.79.227.42:4123 , piece_id=53K6JA7I4GAUEL3ROO65MKFXN7ZSNMOH22UGAXIBRW4WUKN7555A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120653820 (209h32m28.991736865s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.121122984 (209h32m28.992206029s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.121668723 (209h32m28.992751768s) [noise=true , address=145.79.227.46:4603 , piece_id=JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:30.977492703 (209h32m30.848575748s) [upload=183567]: Next returned nil segment Dec 14 09:01:30.977525565 (209h32m30.84860862s) [upload=183567]: Waiting for error group managing segments... Dec 14 09:01:31.270408579 (209h32m31.141491624s) [noise=true , address=145.79.227.42:4123 , piece_id=53K6JA7I4GAUEL3ROO65MKFXN7ZSNMOH22UGAXIBRW4WUKN7555A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.315656118 (209h32m31.186739163s) [noise=true , address=77.90.31.14:4662 , piece_id=TQLMWNW5JAB62NZRPIMTM33N7GSAHZGCU3V7H32LZTSHE2SWMCIQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.342981501 (209h32m31.214064536s) [noise=true , address=77.90.31.21:4878 , piece_id=HF6KZLDP62HJG7MWOAF4TTPXLI7C6CAUX5O4QWT5MP7ZWNZUWDDQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.739040250 (209h32m31.610123295s) [noise=true , address=145.79.227.23:4546 , piece_id=3Q24KFURFMF5POQJ2MNRGWX32E6IKI6XW5XHSLTQ7CCXTEUTGW6A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.800414969 (209h32m31.671498004s) [noise=true , address=145.79.227.47:4265 , piece_id=JLGBSCGU5PK7AAYNRKSJRLMQSKCG4SRM4WSVRALCW7GCA56LRFYQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:32.628091848 (209h32m32.499174883s) [noise=true , address=145.79.227.26:4653 , piece_id=RWUX47DDMWRTEXWKCDN6BF2PGSL6BZAXWQIX5DT547AKMWLNDWYQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:32.629981676 (209h32m32.501064721s) [noise=true , address=145.79.227.49:4337 , piece_id=HUJLCBY3PQDUJ4Y52LESV6CVZ5DMMCV4MO346NTVY3QCSG6NSZCQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.054036275 (209h32m32.92511931s) [noise=true , address=145.79.227.41:4087 , piece_id=66RGE7N2TJ7PSWIBB2UDPIKNZ2ND4BZXFLEXDZWVMVMAYL5PSBGA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.071010907 (209h32m32.942093952s) [noise=true , address=145.79.227.31:4799 , piece_id=L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.120830402 (209h32m32.991913438s) [noise=true , address=145.79.227.46:4601 , piece_id=NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:34.312983241 (209h32m34.184066276s) [noise=true , address=145.79.227.26:4655 , piece_id=6YPJCS6WMI5AYEEESQSWTPK7X4FIUO53V2FONNQYZEKGGBO5PXBA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:34.492220262 (209h32m34.363303307s) [noise=true , address=145.79.227.28:4725 , piece_id=ZDTVQMPD4EUWKRQQVP7PNAQ634L4MQBYJIAFPFSKLQUOTFZHMFPA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:35.590754704 (209h32m35.461837739s) [noise=true , address=145.79.227.46:4603 , piece_id=JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:35.590780002 (209h32m35.461863047s) [seg_pos={0 0}, upload=183567]: Segment reached optimal threshold of 13 pieces. Dec 14 09:01:35.591192537 (209h32m35.462275572s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: read tcp 188.166.225.170:47850->145.79.227.49:4339: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591202196 (209h32m35.462285231s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591227634 (209h32m35.462310669s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: rpc: dial tcp 145.79.227.39:4014: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591241200 (209h32m35.462324245s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591644728 (209h32m35.462727773s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:45550->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591655028 (209h32m35.462738073s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591681468 (209h32m35.462764503s) [seg_pos={0 0}, upload=183567]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 14 09:01:35.591768234 (209h32m35.462851279s) [upload=183567]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 14 09:01:35.591774916 (209h32m35.462857961s) [upload=183567]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 14 09:01:36.093868656 (209h32m35.964951701s) [upload=183567]: Done uploading segments... Dec 14 09:01:36.093895967 (209h32m35.964979012s) [upload=183567]: Upload finished. err: <nil> Duration: 7.954425004s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 7.083327952s trace: https://jaeger.dev.storj.io/trace/9336cdef9668ab3 FAIL (2025-12-14 09:39:02 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 7.083362549s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............XX..............................................] duration: 6.892396867s trace: https://jaeger.dev.storj.io/trace/b3a498e2503e131 FAIL (2025-12-14 09:38:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ) context deadline exceeded; piecestore: (Node ID: 1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX, Piece ID: JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA) context deadline exceeded ======LOGS====== oad=183567]: Starting upload Dec 14 09:01:28.382436168 (209h32m28.253519203s) [upload=183567]: Done starting upload Dec 14 09:01:28.382504638 (209h32m28.253587683s) [upload=183567]: Uploading segments... Dec 14 09:01:28.383014028 (209h32m28.254097073s) [upload=183567]: Got next segment. Inline: false Dec 14 09:01:28.383065245 (209h32m28.254148281s) [upload=183567]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 09:01:28.383070747 (209h32m28.254153782s) [upload=183567]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 09:01:29.117351421 (209h32m28.988434477s) [seg_pos={0 0}, upload=183567]: Begin upload segment... Dec 14 09:01:29.117412969 (209h32m28.988496004s) [seg_pos={0 0}, upload=183567]: Done begin upload segment. Dec 14 09:01:29.117442345 (209h32m28.98852538s) [noise=true , address=145.79.227.26:4655 , piece_id=6YPJCS6WMI5AYEEESQSWTPK7X4FIUO53V2FONNQYZEKGGBO5PXBA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118002081 (209h32m28.989085116s) [noise=true , address=145.79.227.28:4725 , piece_id=ZDTVQMPD4EUWKRQQVP7PNAQ634L4MQBYJIAFPFSKLQUOTFZHMFPA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118088185 (209h32m28.98917122s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118501743 (209h32m28.989584788s) [noise=true , address=77.90.31.21:4878 , piece_id=HF6KZLDP62HJG7MWOAF4TTPXLI7C6CAUX5O4QWT5MP7ZWNZUWDDQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119009108 (209h32m28.990092143s) [noise=true , address=145.79.227.26:4653 , piece_id=RWUX47DDMWRTEXWKCDN6BF2PGSL6BZAXWQIX5DT547AKMWLNDWYQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119070265 (209h32m28.99015331s) [noise=true , address=145.79.227.47:4265 , piece_id=JLGBSCGU5PK7AAYNRKSJRLMQSKCG4SRM4WSVRALCW7GCA56LRFYQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119374144 (209h32m28.990457179s) [noise=true , address=77.90.31.14:4662 , piece_id=TQLMWNW5JAB62NZRPIMTM33N7GSAHZGCU3V7H32LZTSHE2SWMCIQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119462752 (209h32m28.990545797s) [noise=true , address=145.79.227.46:4601 , piece_id=NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119754849 (209h32m28.990837894s) [noise=true , address=145.79.227.41:4087 , piece_id=66RGE7N2TJ7PSWIBB2UDPIKNZ2ND4BZXFLEXDZWVMVMAYL5PSBGA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119815303 (209h32m28.990898338s) [noise=true , address=145.79.227.49:4337 , piece_id=HUJLCBY3PQDUJ4Y52LESV6CVZ5DMMCV4MO346NTVY3QCSG6NSZCQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120045883 (209h32m28.991128918s) [noise=true , address=145.79.227.31:4799 , piece_id=L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120254059 (209h32m28.991337094s) [noise=true , address=145.79.227.23:4546 , piece_id=3Q24KFURFMF5POQJ2MNRGWX32E6IKI6XW5XHSLTQ7CCXTEUTGW6A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120325726 (209h32m28.991408871s) [noise=true , address=145.79.227.42:4123 , piece_id=53K6JA7I4GAUEL3ROO65MKFXN7ZSNMOH22UGAXIBRW4WUKN7555A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120653820 (209h32m28.991736865s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.121122984 (209h32m28.992206029s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.121668723 (209h32m28.992751768s) [noise=true , address=145.79.227.46:4603 , piece_id=JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:30.977492703 (209h32m30.848575748s) [upload=183567]: Next returned nil segment Dec 14 09:01:30.977525565 (209h32m30.84860862s) [upload=183567]: Waiting for error group managing segments... Dec 14 09:01:31.270408579 (209h32m31.141491624s) [noise=true , address=145.79.227.42:4123 , piece_id=53K6JA7I4GAUEL3ROO65MKFXN7ZSNMOH22UGAXIBRW4WUKN7555A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.315656118 (209h32m31.186739163s) [noise=true , address=77.90.31.14:4662 , piece_id=TQLMWNW5JAB62NZRPIMTM33N7GSAHZGCU3V7H32LZTSHE2SWMCIQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.342981501 (209h32m31.214064536s) [noise=true , address=77.90.31.21:4878 , piece_id=HF6KZLDP62HJG7MWOAF4TTPXLI7C6CAUX5O4QWT5MP7ZWNZUWDDQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.739040250 (209h32m31.610123295s) [noise=true , address=145.79.227.23:4546 , piece_id=3Q24KFURFMF5POQJ2MNRGWX32E6IKI6XW5XHSLTQ7CCXTEUTGW6A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.800414969 (209h32m31.671498004s) [noise=true , address=145.79.227.47:4265 , piece_id=JLGBSCGU5PK7AAYNRKSJRLMQSKCG4SRM4WSVRALCW7GCA56LRFYQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:32.628091848 (209h32m32.499174883s) [noise=true , address=145.79.227.26:4653 , piece_id=RWUX47DDMWRTEXWKCDN6BF2PGSL6BZAXWQIX5DT547AKMWLNDWYQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:32.629981676 (209h32m32.501064721s) [noise=true , address=145.79.227.49:4337 , piece_id=HUJLCBY3PQDUJ4Y52LESV6CVZ5DMMCV4MO346NTVY3QCSG6NSZCQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.054036275 (209h32m32.92511931s) [noise=true , address=145.79.227.41:4087 , piece_id=66RGE7N2TJ7PSWIBB2UDPIKNZ2ND4BZXFLEXDZWVMVMAYL5PSBGA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.071010907 (209h32m32.942093952s) [noise=true , address=145.79.227.31:4799 , piece_id=L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.120830402 (209h32m32.991913438s) [noise=true , address=145.79.227.46:4601 , piece_id=NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:34.312983241 (209h32m34.184066276s) [noise=true , address=145.79.227.26:4655 , piece_id=6YPJCS6WMI5AYEEESQSWTPK7X4FIUO53V2FONNQYZEKGGBO5PXBA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:34.492220262 (209h32m34.363303307s) [noise=true , address=145.79.227.28:4725 , piece_id=ZDTVQMPD4EUWKRQQVP7PNAQ634L4MQBYJIAFPFSKLQUOTFZHMFPA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:35.590754704 (209h32m35.461837739s) [noise=true , address=145.79.227.46:4603 , piece_id=JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:35.590780002 (209h32m35.461863047s) [seg_pos={0 0}, upload=183567]: Segment reached optimal threshold of 13 pieces. Dec 14 09:01:35.591192537 (209h32m35.462275572s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: read tcp 188.166.225.170:47850->145.79.227.49:4339: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591202196 (209h32m35.462285231s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591227634 (209h32m35.462310669s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: rpc: dial tcp 145.79.227.39:4014: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591241200 (209h32m35.462324245s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591644728 (209h32m35.462727773s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:45550->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591655028 (209h32m35.462738073s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591681468 (209h32m35.462764503s) [seg_pos={0 0}, upload=183567]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 14 09:01:35.591768234 (209h32m35.462851279s) [upload=183567]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 14 09:01:35.591774916 (209h32m35.462857961s) [upload=183567]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 14 09:01:36.093868656 (209h32m35.964951701s) [upload=183567]: Done uploading segments... Dec 14 09:01:36.093895967 (209h32m35.964979012s) [upload=183567]: Upload finished. err: <nil> Duration: 6.89242451s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.850381606s trace: https://jaeger.dev.storj.io/trace/9336cdef9668ab3 FAIL (2025-12-14 09:38:38 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.850408067s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.175494118s trace: https://jaeger.dev.storj.io/trace/9336cdef9668ab3 FAIL (2025-12-14 09:38:17 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.175522392s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............XX.............................................] duration: 6.309695972s trace: https://jaeger.dev.storj.io/trace/b3a498e2503e131 FAIL (2025-12-14 09:38:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ) context deadline exceeded; piecestore: (Node ID: 1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX, Piece ID: JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA) context deadline exceeded ======LOGS====== oad=183567]: Starting upload Dec 14 09:01:28.382436168 (209h32m28.253519203s) [upload=183567]: Done starting upload Dec 14 09:01:28.382504638 (209h32m28.253587683s) [upload=183567]: Uploading segments... Dec 14 09:01:28.383014028 (209h32m28.254097073s) [upload=183567]: Got next segment. Inline: false Dec 14 09:01:28.383065245 (209h32m28.254148281s) [upload=183567]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 09:01:28.383070747 (209h32m28.254153782s) [upload=183567]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 09:01:29.117351421 (209h32m28.988434477s) [seg_pos={0 0}, upload=183567]: Begin upload segment... Dec 14 09:01:29.117412969 (209h32m28.988496004s) [seg_pos={0 0}, upload=183567]: Done begin upload segment. Dec 14 09:01:29.117442345 (209h32m28.98852538s) [noise=true , address=145.79.227.26:4655 , piece_id=6YPJCS6WMI5AYEEESQSWTPK7X4FIUO53V2FONNQYZEKGGBO5PXBA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118002081 (209h32m28.989085116s) [noise=true , address=145.79.227.28:4725 , piece_id=ZDTVQMPD4EUWKRQQVP7PNAQ634L4MQBYJIAFPFSKLQUOTFZHMFPA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118088185 (209h32m28.98917122s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118501743 (209h32m28.989584788s) [noise=true , address=77.90.31.21:4878 , piece_id=HF6KZLDP62HJG7MWOAF4TTPXLI7C6CAUX5O4QWT5MP7ZWNZUWDDQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119009108 (209h32m28.990092143s) [noise=true , address=145.79.227.26:4653 , piece_id=RWUX47DDMWRTEXWKCDN6BF2PGSL6BZAXWQIX5DT547AKMWLNDWYQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119070265 (209h32m28.99015331s) [noise=true , address=145.79.227.47:4265 , piece_id=JLGBSCGU5PK7AAYNRKSJRLMQSKCG4SRM4WSVRALCW7GCA56LRFYQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119374144 (209h32m28.990457179s) [noise=true , address=77.90.31.14:4662 , piece_id=TQLMWNW5JAB62NZRPIMTM33N7GSAHZGCU3V7H32LZTSHE2SWMCIQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119462752 (209h32m28.990545797s) [noise=true , address=145.79.227.46:4601 , piece_id=NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119754849 (209h32m28.990837894s) [noise=true , address=145.79.227.41:4087 , piece_id=66RGE7N2TJ7PSWIBB2UDPIKNZ2ND4BZXFLEXDZWVMVMAYL5PSBGA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119815303 (209h32m28.990898338s) [noise=true , address=145.79.227.49:4337 , piece_id=HUJLCBY3PQDUJ4Y52LESV6CVZ5DMMCV4MO346NTVY3QCSG6NSZCQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120045883 (209h32m28.991128918s) [noise=true , address=145.79.227.31:4799 , piece_id=L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120254059 (209h32m28.991337094s) [noise=true , address=145.79.227.23:4546 , piece_id=3Q24KFURFMF5POQJ2MNRGWX32E6IKI6XW5XHSLTQ7CCXTEUTGW6A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120325726 (209h32m28.991408871s) [noise=true , address=145.79.227.42:4123 , piece_id=53K6JA7I4GAUEL3ROO65MKFXN7ZSNMOH22UGAXIBRW4WUKN7555A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120653820 (209h32m28.991736865s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.121122984 (209h32m28.992206029s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.121668723 (209h32m28.992751768s) [noise=true , address=145.79.227.46:4603 , piece_id=JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:30.977492703 (209h32m30.848575748s) [upload=183567]: Next returned nil segment Dec 14 09:01:30.977525565 (209h32m30.84860862s) [upload=183567]: Waiting for error group managing segments... Dec 14 09:01:31.270408579 (209h32m31.141491624s) [noise=true , address=145.79.227.42:4123 , piece_id=53K6JA7I4GAUEL3ROO65MKFXN7ZSNMOH22UGAXIBRW4WUKN7555A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.315656118 (209h32m31.186739163s) [noise=true , address=77.90.31.14:4662 , piece_id=TQLMWNW5JAB62NZRPIMTM33N7GSAHZGCU3V7H32LZTSHE2SWMCIQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.342981501 (209h32m31.214064536s) [noise=true , address=77.90.31.21:4878 , piece_id=HF6KZLDP62HJG7MWOAF4TTPXLI7C6CAUX5O4QWT5MP7ZWNZUWDDQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.739040250 (209h32m31.610123295s) [noise=true , address=145.79.227.23:4546 , piece_id=3Q24KFURFMF5POQJ2MNRGWX32E6IKI6XW5XHSLTQ7CCXTEUTGW6A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.800414969 (209h32m31.671498004s) [noise=true , address=145.79.227.47:4265 , piece_id=JLGBSCGU5PK7AAYNRKSJRLMQSKCG4SRM4WSVRALCW7GCA56LRFYQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:32.628091848 (209h32m32.499174883s) [noise=true , address=145.79.227.26:4653 , piece_id=RWUX47DDMWRTEXWKCDN6BF2PGSL6BZAXWQIX5DT547AKMWLNDWYQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:32.629981676 (209h32m32.501064721s) [noise=true , address=145.79.227.49:4337 , piece_id=HUJLCBY3PQDUJ4Y52LESV6CVZ5DMMCV4MO346NTVY3QCSG6NSZCQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.054036275 (209h32m32.92511931s) [noise=true , address=145.79.227.41:4087 , piece_id=66RGE7N2TJ7PSWIBB2UDPIKNZ2ND4BZXFLEXDZWVMVMAYL5PSBGA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.071010907 (209h32m32.942093952s) [noise=true , address=145.79.227.31:4799 , piece_id=L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.120830402 (209h32m32.991913438s) [noise=true , address=145.79.227.46:4601 , piece_id=NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:34.312983241 (209h32m34.184066276s) [noise=true , address=145.79.227.26:4655 , piece_id=6YPJCS6WMI5AYEEESQSWTPK7X4FIUO53V2FONNQYZEKGGBO5PXBA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:34.492220262 (209h32m34.363303307s) [noise=true , address=145.79.227.28:4725 , piece_id=ZDTVQMPD4EUWKRQQVP7PNAQ634L4MQBYJIAFPFSKLQUOTFZHMFPA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:35.590754704 (209h32m35.461837739s) [noise=true , address=145.79.227.46:4603 , piece_id=JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:35.590780002 (209h32m35.461863047s) [seg_pos={0 0}, upload=183567]: Segment reached optimal threshold of 13 pieces. Dec 14 09:01:35.591192537 (209h32m35.462275572s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: read tcp 188.166.225.170:47850->145.79.227.49:4339: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591202196 (209h32m35.462285231s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591227634 (209h32m35.462310669s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: rpc: dial tcp 145.79.227.39:4014: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591241200 (209h32m35.462324245s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591644728 (209h32m35.462727773s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:45550->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591655028 (209h32m35.462738073s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591681468 (209h32m35.462764503s) [seg_pos={0 0}, upload=183567]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 14 09:01:35.591768234 (209h32m35.462851279s) [upload=183567]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 14 09:01:35.591774916 (209h32m35.462857961s) [upload=183567]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 14 09:01:36.093868656 (209h32m35.964951701s) [upload=183567]: Done uploading segments... Dec 14 09:01:36.093895967 (209h32m35.964979012s) [upload=183567]: Upload finished. err: <nil> Duration: 6.309733673s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 6.212695355s trace: https://jaeger.dev.storj.io/trace/9336cdef9668ab3 FAIL (2025-12-14 09:37:13 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.2127504s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............XX............................................] duration: 7.378398098s trace: https://jaeger.dev.storj.io/trace/b3a498e2503e131 FAIL (2025-12-14 09:36:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ) context deadline exceeded; piecestore: (Node ID: 1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX, Piece ID: JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA) context deadline exceeded ======LOGS====== oad=183567]: Starting upload Dec 14 09:01:28.382436168 (209h32m28.253519203s) [upload=183567]: Done starting upload Dec 14 09:01:28.382504638 (209h32m28.253587683s) [upload=183567]: Uploading segments... Dec 14 09:01:28.383014028 (209h32m28.254097073s) [upload=183567]: Got next segment. Inline: false Dec 14 09:01:28.383065245 (209h32m28.254148281s) [upload=183567]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 09:01:28.383070747 (209h32m28.254153782s) [upload=183567]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 09:01:29.117351421 (209h32m28.988434477s) [seg_pos={0 0}, upload=183567]: Begin upload segment... Dec 14 09:01:29.117412969 (209h32m28.988496004s) [seg_pos={0 0}, upload=183567]: Done begin upload segment. Dec 14 09:01:29.117442345 (209h32m28.98852538s) [noise=true , address=145.79.227.26:4655 , piece_id=6YPJCS6WMI5AYEEESQSWTPK7X4FIUO53V2FONNQYZEKGGBO5PXBA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118002081 (209h32m28.989085116s) [noise=true , address=145.79.227.28:4725 , piece_id=ZDTVQMPD4EUWKRQQVP7PNAQ634L4MQBYJIAFPFSKLQUOTFZHMFPA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118088185 (209h32m28.98917122s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118501743 (209h32m28.989584788s) [noise=true , address=77.90.31.21:4878 , piece_id=HF6KZLDP62HJG7MWOAF4TTPXLI7C6CAUX5O4QWT5MP7ZWNZUWDDQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119009108 (209h32m28.990092143s) [noise=true , address=145.79.227.26:4653 , piece_id=RWUX47DDMWRTEXWKCDN6BF2PGSL6BZAXWQIX5DT547AKMWLNDWYQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119070265 (209h32m28.99015331s) [noise=true , address=145.79.227.47:4265 , piece_id=JLGBSCGU5PK7AAYNRKSJRLMQSKCG4SRM4WSVRALCW7GCA56LRFYQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119374144 (209h32m28.990457179s) [noise=true , address=77.90.31.14:4662 , piece_id=TQLMWNW5JAB62NZRPIMTM33N7GSAHZGCU3V7H32LZTSHE2SWMCIQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119462752 (209h32m28.990545797s) [noise=true , address=145.79.227.46:4601 , piece_id=NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119754849 (209h32m28.990837894s) [noise=true , address=145.79.227.41:4087 , piece_id=66RGE7N2TJ7PSWIBB2UDPIKNZ2ND4BZXFLEXDZWVMVMAYL5PSBGA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119815303 (209h32m28.990898338s) [noise=true , address=145.79.227.49:4337 , piece_id=HUJLCBY3PQDUJ4Y52LESV6CVZ5DMMCV4MO346NTVY3QCSG6NSZCQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120045883 (209h32m28.991128918s) [noise=true , address=145.79.227.31:4799 , piece_id=L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120254059 (209h32m28.991337094s) [noise=true , address=145.79.227.23:4546 , piece_id=3Q24KFURFMF5POQJ2MNRGWX32E6IKI6XW5XHSLTQ7CCXTEUTGW6A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120325726 (209h32m28.991408871s) [noise=true , address=145.79.227.42:4123 , piece_id=53K6JA7I4GAUEL3ROO65MKFXN7ZSNMOH22UGAXIBRW4WUKN7555A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120653820 (209h32m28.991736865s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.121122984 (209h32m28.992206029s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.121668723 (209h32m28.992751768s) [noise=true , address=145.79.227.46:4603 , piece_id=JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:30.977492703 (209h32m30.848575748s) [upload=183567]: Next returned nil segment Dec 14 09:01:30.977525565 (209h32m30.84860862s) [upload=183567]: Waiting for error group managing segments... Dec 14 09:01:31.270408579 (209h32m31.141491624s) [noise=true , address=145.79.227.42:4123 , piece_id=53K6JA7I4GAUEL3ROO65MKFXN7ZSNMOH22UGAXIBRW4WUKN7555A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.315656118 (209h32m31.186739163s) [noise=true , address=77.90.31.14:4662 , piece_id=TQLMWNW5JAB62NZRPIMTM33N7GSAHZGCU3V7H32LZTSHE2SWMCIQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.342981501 (209h32m31.214064536s) [noise=true , address=77.90.31.21:4878 , piece_id=HF6KZLDP62HJG7MWOAF4TTPXLI7C6CAUX5O4QWT5MP7ZWNZUWDDQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.739040250 (209h32m31.610123295s) [noise=true , address=145.79.227.23:4546 , piece_id=3Q24KFURFMF5POQJ2MNRGWX32E6IKI6XW5XHSLTQ7CCXTEUTGW6A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.800414969 (209h32m31.671498004s) [noise=true , address=145.79.227.47:4265 , piece_id=JLGBSCGU5PK7AAYNRKSJRLMQSKCG4SRM4WSVRALCW7GCA56LRFYQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:32.628091848 (209h32m32.499174883s) [noise=true , address=145.79.227.26:4653 , piece_id=RWUX47DDMWRTEXWKCDN6BF2PGSL6BZAXWQIX5DT547AKMWLNDWYQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:32.629981676 (209h32m32.501064721s) [noise=true , address=145.79.227.49:4337 , piece_id=HUJLCBY3PQDUJ4Y52LESV6CVZ5DMMCV4MO346NTVY3QCSG6NSZCQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.054036275 (209h32m32.92511931s) [noise=true , address=145.79.227.41:4087 , piece_id=66RGE7N2TJ7PSWIBB2UDPIKNZ2ND4BZXFLEXDZWVMVMAYL5PSBGA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.071010907 (209h32m32.942093952s) [noise=true , address=145.79.227.31:4799 , piece_id=L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.120830402 (209h32m32.991913438s) [noise=true , address=145.79.227.46:4601 , piece_id=NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:34.312983241 (209h32m34.184066276s) [noise=true , address=145.79.227.26:4655 , piece_id=6YPJCS6WMI5AYEEESQSWTPK7X4FIUO53V2FONNQYZEKGGBO5PXBA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:34.492220262 (209h32m34.363303307s) [noise=true , address=145.79.227.28:4725 , piece_id=ZDTVQMPD4EUWKRQQVP7PNAQ634L4MQBYJIAFPFSKLQUOTFZHMFPA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:35.590754704 (209h32m35.461837739s) [noise=true , address=145.79.227.46:4603 , piece_id=JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:35.590780002 (209h32m35.461863047s) [seg_pos={0 0}, upload=183567]: Segment reached optimal threshold of 13 pieces. Dec 14 09:01:35.591192537 (209h32m35.462275572s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: read tcp 188.166.225.170:47850->145.79.227.49:4339: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591202196 (209h32m35.462285231s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591227634 (209h32m35.462310669s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: rpc: dial tcp 145.79.227.39:4014: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591241200 (209h32m35.462324245s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591644728 (209h32m35.462727773s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:45550->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591655028 (209h32m35.462738073s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591681468 (209h32m35.462764503s) [seg_pos={0 0}, upload=183567]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 14 09:01:35.591768234 (209h32m35.462851279s) [upload=183567]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 14 09:01:35.591774916 (209h32m35.462857961s) [upload=183567]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 14 09:01:36.093868656 (209h32m35.964951701s) [upload=183567]: Done uploading segments... Dec 14 09:01:36.093895967 (209h32m35.964979012s) [upload=183567]: Upload finished. err: <nil> Duration: 7.378420189s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 7.292839981s trace: https://jaeger.dev.storj.io/trace/9336cdef9668ab3 FAIL (2025-12-14 09:36:33 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 7.29287603s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 7.424550231s trace: https://jaeger.dev.storj.io/trace/9336cdef9668ab3 FAIL (2025-12-14 09:36:01 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 7.424573045s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............XX...........................................] duration: 7.899166605s trace: https://jaeger.dev.storj.io/trace/b3a498e2503e131 FAIL (2025-12-14 09:35:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, Piece ID: NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ) context deadline exceeded; piecestore: (Node ID: 1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX, Piece ID: JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA) context deadline exceeded ======LOGS====== oad=183567]: Starting upload Dec 14 09:01:28.382436168 (209h32m28.253519203s) [upload=183567]: Done starting upload Dec 14 09:01:28.382504638 (209h32m28.253587683s) [upload=183567]: Uploading segments... Dec 14 09:01:28.383014028 (209h32m28.254097073s) [upload=183567]: Got next segment. Inline: false Dec 14 09:01:28.383065245 (209h32m28.254148281s) [upload=183567]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 09:01:28.383070747 (209h32m28.254153782s) [upload=183567]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 09:01:29.117351421 (209h32m28.988434477s) [seg_pos={0 0}, upload=183567]: Begin upload segment... Dec 14 09:01:29.117412969 (209h32m28.988496004s) [seg_pos={0 0}, upload=183567]: Done begin upload segment. Dec 14 09:01:29.117442345 (209h32m28.98852538s) [noise=true , address=145.79.227.26:4655 , piece_id=6YPJCS6WMI5AYEEESQSWTPK7X4FIUO53V2FONNQYZEKGGBO5PXBA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118002081 (209h32m28.989085116s) [noise=true , address=145.79.227.28:4725 , piece_id=ZDTVQMPD4EUWKRQQVP7PNAQ634L4MQBYJIAFPFSKLQUOTFZHMFPA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118088185 (209h32m28.98917122s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.118501743 (209h32m28.989584788s) [noise=true , address=77.90.31.21:4878 , piece_id=HF6KZLDP62HJG7MWOAF4TTPXLI7C6CAUX5O4QWT5MP7ZWNZUWDDQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119009108 (209h32m28.990092143s) [noise=true , address=145.79.227.26:4653 , piece_id=RWUX47DDMWRTEXWKCDN6BF2PGSL6BZAXWQIX5DT547AKMWLNDWYQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119070265 (209h32m28.99015331s) [noise=true , address=145.79.227.47:4265 , piece_id=JLGBSCGU5PK7AAYNRKSJRLMQSKCG4SRM4WSVRALCW7GCA56LRFYQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119374144 (209h32m28.990457179s) [noise=true , address=77.90.31.14:4662 , piece_id=TQLMWNW5JAB62NZRPIMTM33N7GSAHZGCU3V7H32LZTSHE2SWMCIQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119462752 (209h32m28.990545797s) [noise=true , address=145.79.227.46:4601 , piece_id=NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119754849 (209h32m28.990837894s) [noise=true , address=145.79.227.41:4087 , piece_id=66RGE7N2TJ7PSWIBB2UDPIKNZ2ND4BZXFLEXDZWVMVMAYL5PSBGA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.119815303 (209h32m28.990898338s) [noise=true , address=145.79.227.49:4337 , piece_id=HUJLCBY3PQDUJ4Y52LESV6CVZ5DMMCV4MO346NTVY3QCSG6NSZCQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120045883 (209h32m28.991128918s) [noise=true , address=145.79.227.31:4799 , piece_id=L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120254059 (209h32m28.991337094s) [noise=true , address=145.79.227.23:4546 , piece_id=3Q24KFURFMF5POQJ2MNRGWX32E6IKI6XW5XHSLTQ7CCXTEUTGW6A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120325726 (209h32m28.991408871s) [noise=true , address=145.79.227.42:4123 , piece_id=53K6JA7I4GAUEL3ROO65MKFXN7ZSNMOH22UGAXIBRW4WUKN7555A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.120653820 (209h32m28.991736865s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.121122984 (209h32m28.992206029s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:29.121668723 (209h32m28.992751768s) [noise=true , address=145.79.227.46:4603 , piece_id=JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA, seg_pos={0 0}, upload=183567]: Uploading piece... Dec 14 09:01:30.977492703 (209h32m30.848575748s) [upload=183567]: Next returned nil segment Dec 14 09:01:30.977525565 (209h32m30.84860862s) [upload=183567]: Waiting for error group managing segments... Dec 14 09:01:31.270408579 (209h32m31.141491624s) [noise=true , address=145.79.227.42:4123 , piece_id=53K6JA7I4GAUEL3ROO65MKFXN7ZSNMOH22UGAXIBRW4WUKN7555A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.315656118 (209h32m31.186739163s) [noise=true , address=77.90.31.14:4662 , piece_id=TQLMWNW5JAB62NZRPIMTM33N7GSAHZGCU3V7H32LZTSHE2SWMCIQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.342981501 (209h32m31.214064536s) [noise=true , address=77.90.31.21:4878 , piece_id=HF6KZLDP62HJG7MWOAF4TTPXLI7C6CAUX5O4QWT5MP7ZWNZUWDDQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.739040250 (209h32m31.610123295s) [noise=true , address=145.79.227.23:4546 , piece_id=3Q24KFURFMF5POQJ2MNRGWX32E6IKI6XW5XHSLTQ7CCXTEUTGW6A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:31.800414969 (209h32m31.671498004s) [noise=true , address=145.79.227.47:4265 , piece_id=JLGBSCGU5PK7AAYNRKSJRLMQSKCG4SRM4WSVRALCW7GCA56LRFYQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:32.628091848 (209h32m32.499174883s) [noise=true , address=145.79.227.26:4653 , piece_id=RWUX47DDMWRTEXWKCDN6BF2PGSL6BZAXWQIX5DT547AKMWLNDWYQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:32.629981676 (209h32m32.501064721s) [noise=true , address=145.79.227.49:4337 , piece_id=HUJLCBY3PQDUJ4Y52LESV6CVZ5DMMCV4MO346NTVY3QCSG6NSZCQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.054036275 (209h32m32.92511931s) [noise=true , address=145.79.227.41:4087 , piece_id=66RGE7N2TJ7PSWIBB2UDPIKNZ2ND4BZXFLEXDZWVMVMAYL5PSBGA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.071010907 (209h32m32.942093952s) [noise=true , address=145.79.227.31:4799 , piece_id=L4LKCBHWWCXKZE6BKT7WCVRUTITVBW3WBJEPTMMROQVNG3FGXKBQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:33.120830402 (209h32m32.991913438s) [noise=true , address=145.79.227.46:4601 , piece_id=NOJS2U5MHR3VOKC7DK7UHONLUYFC5BAVFDJMXYPPDJ444URVMRQQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:34.312983241 (209h32m34.184066276s) [noise=true , address=145.79.227.26:4655 , piece_id=6YPJCS6WMI5AYEEESQSWTPK7X4FIUO53V2FONNQYZEKGGBO5PXBA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:34.492220262 (209h32m34.363303307s) [noise=true , address=145.79.227.28:4725 , piece_id=ZDTVQMPD4EUWKRQQVP7PNAQ634L4MQBYJIAFPFSKLQUOTFZHMFPA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:35.590754704 (209h32m35.461837739s) [noise=true , address=145.79.227.46:4603 , piece_id=JI2FY675AKWU3RCNNNTSC7LIKOVQHGLIMXGDXUEK3M6AUT3XHIMA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: <nil> Dec 14 09:01:35.590780002 (209h32m35.461863047s) [seg_pos={0 0}, upload=183567]: Segment reached optimal threshold of 13 pieces. Dec 14 09:01:35.591192537 (209h32m35.462275572s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: read tcp 188.166.225.170:47850->145.79.227.49:4339: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591202196 (209h32m35.462285231s) [noise=true , address=145.79.227.49:4339 , piece_id=L4LO6LUYJCK4XVYKHVT6XFC7MA6S6HEWVF2TH3WQE53G6NRY3T4A, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591227634 (209h32m35.462310669s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: rpc: dial tcp 145.79.227.39:4014: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591241200 (209h32m35.462324245s) [noise=true , address=145.79.227.39:4014 , piece_id=MX5JWHFN2D6HINNXBVO4KLAJ52ZUGWSXEIDFBK3HUMKJNCSHRNBQ, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591644728 (209h32m35.462727773s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:45550->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 14 09:01:35.591655028 (209h32m35.462738073s) [noise=true , address=145.79.227.38:4977 , piece_id=MSLUWRN6OD5UTSGZODW3FNXADA4FF6VPTNT22DIFJXR75L2GFKAA, seg_pos={0 0}, upload=183567]: Long tail cancellation due to optimal threshold Dec 14 09:01:35.591681468 (209h32m35.462764503s) [seg_pos={0 0}, upload=183567]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 14 09:01:35.591768234 (209h32m35.462851279s) [upload=183567]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 14 09:01:35.591774916 (209h32m35.462857961s) [upload=183567]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 14 09:01:36.093868656 (209h32m35.964951701s) [upload=183567]: Done uploading segments... Dec 14 09:01:36.093895967 (209h32m35.964979012s) [upload=183567]: Upload finished. err: <nil> Duration: 7.899198204s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)