Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [X...X...X...........X..X.X...........X....XX.XXX..XXX.X....X] duration:
30.003145814s trace:
https://jaeger.dev.storj.io/trace/4db21358fee849a2
FAIL (2025-12-26 14:36:21 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: KYH425DXJLSTSZCZHTZQS66KILAKNBRD5XWYWQFYK2OC7RFF3GHA) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: XQCQKDB7LBYYQVV2CGLF5K3WPT2VJHNL7HIS6EX4HOTWSATPVPQQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: HXM3C7EFKNKM5AC63SLYJWM7BRXWGD3B3UAUEOP3KNZD36JYHVYA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: QITLYC5DTDAPDLRXXTSNEARXMFM6G4MEEP5DO722CJ6BAOE4Z35Q) context deadline exceeded
======LOGS======
Dec 26 14:35:51.708129116 (503h6m51.579212151s) [upload=441509]: Starting upload
Dec 26 14:35:51.708220490 (503h6m51.579303535s) [upload=441509]: Done starting upload
Dec 26 14:35:51.708331171 (503h6m51.579414206s) [upload=441509]: Uploading segments...
Dec 26 14:35:51.708517366 (503h6m51.579600401s) [upload=441509]: Got next segment. Inline: false
Dec 26 14:35:51.708568934 (503h6m51.579651969s) [upload=441509]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 14:35:51.708577770 (503h6m51.579660805s) [upload=441509]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 14:35:51.956827684 (503h6m51.827910719s) [seg_pos={0 0}, upload=441509]: Begin upload segment...
Dec 26 14:35:51.956929187 (503h6m51.828012232s) [seg_pos={0 0}, upload=441509]: Done begin upload segment.
Dec 26 14:35:51.957181748 (503h6m51.828264793s) [noise=true , address=145.79.227.35:4483 , piece_id=CX6YIAEX6YD4SPM3GTB3MP4I22LQS3Q4JGYCPP23WNO5K6SW7QZQ, seg_pos={0 0}, upload=441509]: Uploading piece...
Dec 26 14:35:51.957245569 (503h6m51.828328614s) [noise=true , address=45.91.116.10:4084 , piece_id=5LO2KVNTZB5HBWBZIZ77EKX5XUEMSXLUPBLWKPH52XZRRPB77XXA, seg_pos={0 0}, upload=441509]: Uploading piece...
Dec 26 14:35:51.958269689 (503h6m51.829352734s) [noise=true , address=145.79.227.50:4373 , piece_id=JXHVDFOCN54NVSPOSNQJAUIXLAZWURAWJOIELKEQVWDE7VYATEQA, seg_pos={0 0}, upload=441509]: Uploading piece...
Dec 26 14:35:51.959289532 (503h6m51.830372567s) [noise=true , address=45.91.116.11:4119 , piece_id=7YUH7NACQHAKCKSTH3FVVV5IV4VIIXLVFQ7FXNC7L5LBHJ2UCOVQ, seg_pos={0 0}, upload=441509]: Uploading piece...
Dec 26 14:35:51.959936723 (503h6m51.831019768s) [noise=true , address=45.91.116.14:4192 , piece_id=QITLYC5DTDAPDLRXXTSNEARXMFM6G4MEEP5DO722CJ6BAOE4Z35Q, seg_pos={0 0}, upload=441509]: Uploading piece...
Dec 26 14:35:51.960609876 (503h6m51.831692921s) [noise=true , address=145.79.227.13:4804 , piece_id=HHASV3ENCLJSX5WDVYH3RYCJCMQWX7TASCYJSLPW3IZSOZBUUCAQ, seg_pos={0 0}, upload=441509]: Uploading piece...
Dec 26 14:35:51.961391844 (503h6m51.832474889s) [noise=true , address=45.91.116.9:4263 , piece_id=KYH425DXJLSTSZCZHTZQS66KILAKNBRD5XWYWQFYK2OC7RFF3GHA, seg_pos={0 0}, upload=441509]: Uploading piece...
Dec 26 14:35:51.962263294 (503h6m51.833346339s) [noise=true , address=77.90.31.13:4627 , piece_id=P4MGSUKHF6KVV4OYWRS24AODLW4KI27QFA2IC3YKT7IPCRTAYBGQ, seg_pos={0 0}, upload=441509]: Uploading piece...
Dec 26 14:35:51.962853698 (503h6m51.833936743s) [noise=true , address=45.91.116.13:4154 , piece_id=XQCQKDB7LBYYQVV2CGLF5K3WPT2VJHNL7HIS6EX4HOTWSATPVPQQ, seg_pos={0 0}, upload=441509]: Uploading piece...
Dec 26 14:35:51.963666316 (503h6m51.834749351s) [noise=true , address=145.79.227.52:4445 , piece_id=37ZQGT2T6NWQS2NVMC5UABQXKDN34FDBIWZMUM3QDYN3YT6W76CA, seg_pos={0 0}, upload=441509]: Uploading piece...
Dec 26 14:35:51.964521284 (503h6m51.835604339s) [noise=true , address=145.79.227.7:4294 , piece_id=IGXXZE7EQERKA7HCI6GN4ZG2M22HHZLIWAPXBFWCURXAGC4ZYC5Q, seg_pos={0 0}, upload=441509]: Uploading piece...
Dec 26 14:35:51.965223490 (503h6m51.836306546s) [noise=true , address=145.79.227.45:4229 , piece_id=RVABJZBTWKAJJK3KF5OJXZ2THQCYZNDKPKJNJJV64IQYKRZMGDBA, seg_pos={0 0}, upload=441509]: Uploading piece...
Dec 26 14:35:51.966178970 (503h6m51.837262015s) [noise=true , address=145.79.227.17:4970 , piece_id=OBK2AIDV7IC77XRHLBQVJFL4LZN7FJUA4SFGGBRVR64ZIVLKV4QA, seg_pos={0 0}, upload=441509]: Uploading piece...
Dec 26 14:35:51.968036597 (503h6m51.839119652s) [noise=true , address=145.79.227.14:4840 , piece_id=XWF4UO7ITHM6C6MYDKQQQXBHRM5YAJT4WGZ5DSXX7HQH5OCLZHQA, seg_pos={0 0}, upload=441509]: Uploading piece...
Dec 26 14:35:51.970226327 (503h6m51.841309372s) [noise=true , address=45.91.116.6:4975 , piece_id=HXM3C7EFKNKM5AC63SLYJWM7BRXWGD3B3UAUEOP3KNZD36JYHVYA, seg_pos={0 0}, upload=441509]: Uploading piece...
Dec 26 14:35:51.972931468 (503h6m51.844014503s) [noise=true , address=45.91.116.7:4010 , piece_id=LRCZSMAV7BT3W67ASSIMRC7RW7PLFTO524SUY4NWAZY2P5SKQPIQ, seg_pos={0 0}, upload=441509]: Uploading piece...
Dec 26 14:35:53.662220654 (503h6m53.533303699s) [upload=441509]: Next returned nil segment
Dec 26 14:35:53.662237686 (503h6m53.533320731s) [upload=441509]: Waiting for error group managing segments...
Dec 26 14:35:53.945679881 (503h6m53.816762926s) [noise=true , address=145.79.227.50:4373 , piece_id=JXHVDFOCN54NVSPOSNQJAUIXLAZWURAWJOIELKEQVWDE7VYATEQA, seg_pos={0 0}, upload=441509]: Done uploading piece. err: <nil>
Dec 26 14:35:53.955163951 (503h6m53.826246996s) [noise=true , address=145.79.227.35:4483 , piece_id=CX6YIAEX6YD4SPM3GTB3MP4I22LQS3Q4JGYCPP23WNO5K6SW7QZQ, seg_pos={0 0}, upload=441509]: Done uploading piece. err: <nil>
Dec 26 14:35:53.955917525 (503h6m53.82700056s) [noise=true , address=145.79.227.45:4229 , piece_id=RVABJZBTWKAJJK3KF5OJXZ2THQCYZNDKPKJNJJV64IQYKRZMGDBA, seg_pos={0 0}, upload=441509]: Done uploading piece. err: <nil>
Dec 26 14:35:53.966341525 (503h6m53.83742456s) [noise=true , address=145.79.227.13:4804 , piece_id=HHASV3ENCLJSX5WDVYH3RYCJCMQWX7TASCYJSLPW3IZSOZBUUCAQ, seg_pos={0 0}, upload=441509]: Done uploading piece. err: <nil>
Dec 26 14:35:53.967435217 (503h6m53.838518262s) [noise=true , address=145.79.227.52:4445 , piece_id=37ZQGT2T6NWQS2NVMC5UABQXKDN34FDBIWZMUM3QDYN3YT6W76CA, seg_pos={0 0}, upload=441509]: Done uploading piece. err: <nil>
Dec 26 14:35:53.975284394 (503h6m53.846367429s) [noise=true , address=145.79.227.7:4294 , piece_id=IGXXZE7EQERKA7HCI6GN4ZG2M22HHZLIWAPXBFWCURXAGC4ZYC5Q, seg_pos={0 0}, upload=441509]: Done uploading piece. err: <nil>
Dec 26 14:35:53.976586394 (503h6m53.847669429s) [noise=true , address=145.79.227.17:4970 , piece_id=OBK2AIDV7IC77XRHLBQVJFL4LZN7FJUA4SFGGBRVR64ZIVLKV4QA, seg_pos={0 0}, upload=441509]: Done uploading piece. err: <nil>
Dec 26 14:35:53.982741694 (503h6m53.853824749s) [noise=true , address=145.79.227.14:4840 , piece_id=XWF4UO7ITHM6C6MYDKQQQXBHRM5YAJT4WGZ5DSXX7HQH5OCLZHQA, seg_pos={0 0}, upload=441509]: Done uploading piece. err: <nil>
Dec 26 14:35:54.266102615 (503h6m54.13718566s) [noise=true , address=45.91.116.13:4154 , piece_id=XQCQKDB7LBYYQVV2CGLF5K3WPT2VJHNL7HIS6EX4HOTWSATPVPQQ, seg_pos={0 0}, upload=441509]: Done uploading piece. err: <nil>
Dec 26 14:35:54.324642546 (503h6m54.195725591s) [noise=true , address=45.91.116.6:4975 , piece_id=HXM3C7EFKNKM5AC63SLYJWM7BRXWGD3B3UAUEOP3KNZD36JYHVYA, seg_pos={0 0}, upload=441509]: Done uploading piece. err: <nil>
Dec 26 14:35:54.342833935 (503h6m54.21391698s) [noise=true , address=45.91.116.11:4119 , piece_id=7YUH7NACQHAKCKSTH3FVVV5IV4VIIXLVFQ7FXNC7L5LBHJ2UCOVQ, seg_pos={0 0}, upload=441509]: Done uploading piece. err: <nil>
Dec 26 14:35:54.650189667 (503h6m54.521272702s) [noise=true , address=45.91.116.14:4192 , piece_id=QITLYC5DTDAPDLRXXTSNEARXMFM6G4MEEP5DO722CJ6BAOE4Z35Q, seg_pos={0 0}, upload=441509]: Done uploading piece. err: <nil>
Dec 26 14:35:54.961019614 (503h6m54.832102649s) [noise=true , address=45.91.116.9:4263 , piece_id=KYH425DXJLSTSZCZHTZQS66KILAKNBRD5XWYWQFYK2OC7RFF3GHA, seg_pos={0 0}, upload=441509]: Done uploading piece. err: <nil>
Dec 26 14:35:54.961042608 (503h6m54.832125653s) [seg_pos={0 0}, upload=441509]: Segment reached optimal threshold of 13 pieces.
Dec 26 14:35:54.961506060 (503h6m54.832589095s) [noise=true , address=77.90.31.13:4627 , piece_id=P4MGSUKHF6KVV4OYWRS24AODLW4KI27QFA2IC3YKT7IPCRTAYBGQ, seg_pos={0 0}, upload=441509]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF
Dec 26 14:35:54.961513634 (503h6m54.832596669s) [noise=true , address=77.90.31.13:4627 , piece_id=P4MGSUKHF6KVV4OYWRS24AODLW4KI27QFA2IC3YKT7IPCRTAYBGQ, seg_pos={0 0}, upload=441509]: Long tail cancellation due to optimal threshold
Dec 26 14:35:54.961642129 (503h6m54.832725164s) [noise=true , address=45.91.116.10:4084 , piece_id=5LO2KVNTZB5HBWBZIZ77EKX5XUEMSXLUPBLWKPH52XZRRPB77XXA, seg_pos={0 0}, upload=441509]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: read tcp 188.166.225.170:58860->45.91.116.10:4084: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 14:35:54.961658330 (503h6m54.832741365s) [noise=true , address=45.91.116.10:4084 , piece_id=5LO2KVNTZB5HBWBZIZ77EKX5XUEMSXLUPBLWKPH52XZRRPB77XXA, seg_pos={0 0}, upload=441509]: Long tail cancellation due to optimal threshold
Dec 26 14:35:54.961696473 (503h6m54.832779508s) [noise=true , address=45.91.116.7:4010 , piece_id=LRCZSMAV7BT3W67ASSIMRC7RW7PLFTO524SUY4NWAZY2P5SKQPIQ, seg_pos={0 0}, upload=441509]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: read tcp 188.166.225.170:36742->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 14:35:54.961705490 (503h6m54.832788525s) [noise=true , address=45.91.116.7:4010 , piece_id=LRCZSMAV7BT3W67ASSIMRC7RW7PLFTO524SUY4NWAZY2P5SKQPIQ, seg_pos={0 0}, upload=441509]: Long tail cancellation due to optimal threshold
Dec 26 14:35:54.961722582 (503h6m54.832805617s) [seg_pos={0 0}, upload=441509]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 14:35:54.961789640 (503h6m54.832872675s) [upload=441509]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:35:54.961825759 (503h6m54.832908794s) [upload=441509]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:35:55.259340241 (503h6m55.130423286s) [upload=441509]: Done uploading segments...
Dec 26 14:35:55.259364967 (503h6m55.130448012s) [upload=441509]: Upload finished. err: <nil>
Duration: 30.003172414s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
12.20582436s trace:
https://jaeger.dev.storj.io/trace/5b63846e68ea1b18
FAIL (2025-12-26 14:36:02 UTC)
orbiter warning: context deadline exceeded
Duration: 12.205843356s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
13.770178335s trace:
https://jaeger.dev.storj.io/trace/3ccdb98964ad100c
FAIL (2025-12-26 14:35:57 UTC)
orbiter warning: context deadline exceeded
Duration: 13.770255451s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
55 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXX.XXX.XXXXX.XXXXXXX.XXXXXXXXX.XXXXX] duration:
5.00079139s trace:
https://jaeger.dev.storj.io/trace/21a56756bf5012f5
FAIL (2025-12-26 14:35:55 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000809765s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
18 out of 60 is failed (threshold=3) [.....X..........X..X..X.X.X....X.....XXX.XX.XX.....X...XX.X.] duration:
48.257564069s trace:
https://jaeger.dev.storj.io/trace/7e11b9275a734af0
FAIL (2025-12-26 14:35:42 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
26 14:33:55.235683223 (503h4m55.106766268s) [noise=true , address=77.90.31.13:4650 , piece_id=LNQNQRIHW4TRCOEZFGANMG7IWB5O6MHVTS4BL6UTI34AL5OIOLKA, seg_pos={0 0}, upload=441482]: Uploading piece...
Dec 26 14:33:55.236058588 (503h4m55.107141633s) [noise=true , address=145.79.227.30:4780 , piece_id=NDJPRC7S4BLPP757YCO3RPHXVPPBP7333K4NRPE6GBDBKTC5XB6A, seg_pos={0 0}, upload=441482]: Uploading piece...
Dec 26 14:33:55.236348910 (503h4m55.107431955s) [noise=true , address=45.91.116.13:4160 , piece_id=BHHK2DRNWQNJEYF5APSN6RPBSRC6GLOAVDMHC3QAFE3RKGTWMFNQ, seg_pos={0 0}, upload=441482]: Uploading piece...
Dec 26 14:33:55.236652318 (503h4m55.107735353s) [noise=true , address=145.79.227.44:4197 , piece_id=SAKARE65PJ3VL56Z272CJLUQ3XOSHSV4VHPZTCCOLEGUXBKYUGMQ, seg_pos={0 0}, upload=441482]: Uploading piece...
Dec 26 14:33:55.236937321 (503h4m55.108020356s) [noise=true , address=77.90.31.16:4743 , piece_id=USSS6LMMUVCNGYVZMBCXC72OH5SPJKF64P5CVT65BIFWCRPC6NAQ, seg_pos={0 0}, upload=441482]: Uploading piece...
Dec 26 14:33:55.237356940 (503h4m55.108439985s) [noise=true , address=64.49.13.138:4489 , piece_id=FYQKAISGAQRIXFB2IIUIPYOY5NDF5YEHWSFZ5FPS7WQDXFZ6BWFQ, seg_pos={0 0}, upload=441482]: Uploading piece...
Dec 26 14:33:55.237659506 (503h4m55.108742551s) [noise=true , address=145.79.227.40:4059 , piece_id=XLYE4JHTS56AKAG2SI7RWMDC6IGROOFNQA6NO6EZG6DZADGN65DQ, seg_pos={0 0}, upload=441482]: Uploading piece...
Dec 26 14:33:55.237988353 (503h4m55.109071398s) [noise=true , address=77.90.31.17:4769 , piece_id=WH7Z3ZN67MRXPVTDIVT57CRPMUXVVQMSTFVICQB5M3MEBV7LALTQ, seg_pos={0 0}, upload=441482]: Uploading piece...
Dec 26 14:33:55.238279647 (503h4m55.109362682s) [noise=true , address=77.90.31.21:4903 , piece_id=F5WG3S4YBD7NQ2VCA3HWVOYBRRHPEYRITFTEKPHINL7X6Q5UGY6A, seg_pos={0 0}, upload=441482]: Uploading piece...
Dec 26 14:33:55.238702262 (503h4m55.109785307s) [noise=true , address=77.90.31.14:4679 , piece_id=IXEPYMDUICJM6YAPX5DDQETYZQ6ZJG43FTAGYKX2MAX2G2QWGVLA, seg_pos={0 0}, upload=441482]: Uploading piece...
Dec 26 14:33:55.239070814 (503h4m55.110153849s) [noise=true , address=145.79.227.52:4469 , piece_id=EOIJ6Z4OKARUW3KNGGNEYBQ2QRLRSGEJTPZ7TZBRV4NBAJHAQ6GQ, seg_pos={0 0}, upload=441482]: Uploading piece...
Dec 26 14:33:55.240243927 (503h4m55.111326972s) [noise=true , address=77.90.31.12:4607 , piece_id=HFZCUOPCYLCU6PQGH2UHKIHIVLORJG4MTM4CCWA7XJ6H6JBAPFYQ, seg_pos={0 0}, upload=441482]: Uploading piece...
Dec 26 14:33:55.240758477 (503h4m55.111841512s) [noise=true , address=77.90.31.15:4704 , piece_id=IW2IDZMLW3WLTTUIVGJMB4CTIL7U5INVUNYP7AV4QBERU6J6NNCA, seg_pos={0 0}, upload=441482]: Uploading piece...
Dec 26 14:33:55.242161258 (503h4m55.113244293s) [noise=true , address=77.90.31.11:4564 , piece_id=GOJICK6HKWQKBNICAUWE7KWHZCPSTVRUTNQOHVYK2I6B7Z46PS7A, seg_pos={0 0}, upload=441482]: Uploading piece...
Dec 26 14:33:55.243174468 (503h4m55.114257513s) [noise=true , address=64.49.13.155:4920 , piece_id=RM6ZUHHBVOFC7REQZ5A2IOJK2ZNOL2XYFIDXQIBSJBKPHW3NJSLQ, seg_pos={0 0}, upload=441482]: Uploading piece...
Dec 26 14:33:55.244432463 (503h4m55.115515509s) [noise=true , address=77.90.31.18:4834 , piece_id=6ONKW2LMMA5HVDGAW2ZY5KIU5BOL6PRK6F2XQ6DZJMIG2B4ZB2ZA, seg_pos={0 0}, upload=441482]: Uploading piece...
Dec 26 14:33:56.587466035 (503h4m56.45854908s) [upload=441482]: Next returned nil segment
Dec 26 14:33:56.587504639 (503h4m56.458587674s) [upload=441482]: Waiting for error group managing segments...
Dec 26 14:33:56.941402434 (503h4m56.812485479s) [noise=true , address=64.49.13.155:4920 , piece_id=RM6ZUHHBVOFC7REQZ5A2IOJK2ZNOL2XYFIDXQIBSJBKPHW3NJSLQ, seg_pos={0 0}, upload=441482]: Done uploading piece. err: <nil>
Dec 26 14:33:57.105592201 (503h4m56.976675236s) [noise=true , address=145.79.227.40:4059 , piece_id=XLYE4JHTS56AKAG2SI7RWMDC6IGROOFNQA6NO6EZG6DZADGN65DQ, seg_pos={0 0}, upload=441482]: Done uploading piece. err: <nil>
Dec 26 14:33:57.109300512 (503h4m56.980383548s) [noise=true , address=145.79.227.52:4469 , piece_id=EOIJ6Z4OKARUW3KNGGNEYBQ2QRLRSGEJTPZ7TZBRV4NBAJHAQ6GQ, seg_pos={0 0}, upload=441482]: Done uploading piece. err: <nil>
Dec 26 14:33:57.310829930 (503h4m57.181912965s) [noise=true , address=77.90.31.15:4704 , piece_id=IW2IDZMLW3WLTTUIVGJMB4CTIL7U5INVUNYP7AV4QBERU6J6NNCA, seg_pos={0 0}, upload=441482]: Done uploading piece. err: <nil>
Dec 26 14:33:57.325869969 (503h4m57.196953034s) [noise=true , address=77.90.31.18:4834 , piece_id=6ONKW2LMMA5HVDGAW2ZY5KIU5BOL6PRK6F2XQ6DZJMIG2B4ZB2ZA, seg_pos={0 0}, upload=441482]: Done uploading piece. err: <nil>
Dec 26 14:33:57.434979604 (503h4m57.306062639s) [noise=true , address=77.90.31.17:4769 , piece_id=WH7Z3ZN67MRXPVTDIVT57CRPMUXVVQMSTFVICQB5M3MEBV7LALTQ, seg_pos={0 0}, upload=441482]: Done uploading piece. err: <nil>
Dec 26 14:33:57.498015707 (503h4m57.369098742s) [noise=true , address=45.91.116.13:4160 , piece_id=BHHK2DRNWQNJEYF5APSN6RPBSRC6GLOAVDMHC3QAFE3RKGTWMFNQ, seg_pos={0 0}, upload=441482]: Done uploading piece. err: <nil>
Dec 26 14:33:57.795622655 (503h4m57.6667057s) [noise=true , address=77.90.31.13:4650 , piece_id=LNQNQRIHW4TRCOEZFGANMG7IWB5O6MHVTS4BL6UTI34AL5OIOLKA, seg_pos={0 0}, upload=441482]: Done uploading piece. err: <nil>
Dec 26 14:33:57.958552623 (503h4m57.829635658s) [noise=true , address=77.90.31.11:4564 , piece_id=GOJICK6HKWQKBNICAUWE7KWHZCPSTVRUTNQOHVYK2I6B7Z46PS7A, seg_pos={0 0}, upload=441482]: Done uploading piece. err: <nil>
Dec 26 14:33:57.984892960 (503h4m57.855976005s) [noise=true , address=77.90.31.21:4903 , piece_id=F5WG3S4YBD7NQ2VCA3HWVOYBRRHPEYRITFTEKPHINL7X6Q5UGY6A, seg_pos={0 0}, upload=441482]: Done uploading piece. err: <nil>
Dec 26 14:33:58.006177078 (503h4m57.877260113s) [noise=true , address=145.79.227.44:4197 , piece_id=SAKARE65PJ3VL56Z272CJLUQ3XOSHSV4VHPZTCCOLEGUXBKYUGMQ, seg_pos={0 0}, upload=441482]: Done uploading piece. err: <nil>
Dec 26 14:33:58.040690618 (503h4m57.911773653s) [noise=true , address=45.91.116.10:4114 , piece_id=63CCS6OE5TP7RI6QMO3VPGE5JI4T3EOXHI5ZR5ZGB3I75WECUYSQ, seg_pos={0 0}, upload=441482]: Done uploading piece. err: <nil>
Dec 26 14:33:58.063917928 (503h4m57.935000963s) [noise=true , address=145.79.227.30:4780 , piece_id=NDJPRC7S4BLPP757YCO3RPHXVPPBP7333K4NRPE6GBDBKTC5XB6A, seg_pos={0 0}, upload=441482]: Done uploading piece. err: <nil>
Dec 26 14:33:58.102298952 (503h4m57.973381997s) [noise=true , address=77.90.31.16:4743 , piece_id=USSS6LMMUVCNGYVZMBCXC72OH5SPJKF64P5CVT65BIFWCRPC6NAQ, seg_pos={0 0}, upload=441482]: Done uploading piece. err: <nil>
Dec 26 14:33:58.102329440 (503h4m57.973412476s) [seg_pos={0 0}, upload=441482]: Segment reached optimal threshold of 14 pieces.
Dec 26 14:33:58.102543318 (503h4m57.973626363s) [noise=true , address=77.90.31.14:4679 , piece_id=IXEPYMDUICJM6YAPX5DDQETYZQ6ZJG43FTAGYKX2MAX2G2QWGVLA, seg_pos={0 0}, upload=441482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k29MGpez3ByMHnG8qGBm9iPwo3VrKUGDHpz9CqGEGNn47rxDx): protocol: expected piece hash; context canceled; EOF
Dec 26 14:33:58.102550972 (503h4m57.973634007s) [noise=true , address=77.90.31.14:4679 , piece_id=IXEPYMDUICJM6YAPX5DDQETYZQ6ZJG43FTAGYKX2MAX2G2QWGVLA, seg_pos={0 0}, upload=441482]: Long tail cancellation due to optimal threshold
Dec 26 14:33:58.102670821 (503h4m57.973753876s) [noise=true , address=77.90.31.12:4607 , piece_id=HFZCUOPCYLCU6PQGH2UHKIHIVLORJG4MTM4CCWA7XJ6H6JBAPFYQ, seg_pos={0 0}, upload=441482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm): protocol: expected piece hash; context canceled; EOF
Dec 26 14:33:58.102680829 (503h4m57.973763875s) [noise=true , address=77.90.31.12:4607 , piece_id=HFZCUOPCYLCU6PQGH2UHKIHIVLORJG4MTM4CCWA7XJ6H6JBAPFYQ, seg_pos={0 0}, upload=441482]: Long tail cancellation due to optimal threshold
Dec 26 14:33:58.102713862 (503h4m57.973796908s) [noise=true , address=64.49.13.138:4489 , piece_id=FYQKAISGAQRIXFB2IIUIPYOY5NDF5YEHWSFZ5FPS7WQDXFZ6BWFQ, seg_pos={0 0}, upload=441482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CiiMtdAoMhPZwSqwraaQcrrx4vZw6L74vadw712VQbRLpNzKN): protocol: expected piece hash; context canceled; EOF
Dec 26 14:33:58.102720966 (503h4m57.973804011s) [noise=true , address=64.49.13.138:4489 , piece_id=FYQKAISGAQRIXFB2IIUIPYOY5NDF5YEHWSFZ5FPS7WQDXFZ6BWFQ, seg_pos={0 0}, upload=441482]: Long tail cancellation due to optimal threshold
Dec 26 14:33:58.102875280 (503h4m57.973958335s) [noise=true , address=64.49.13.169:4711 , piece_id=N7F5HCN5E32LOF4WTQSZUIT7Z6VACAEYNDWVFHIYFSBKRIT3QXVA, seg_pos={0 0}, upload=441482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:56296->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:56296->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56296->64.49.13.169:4711: use of closed network connection
Dec 26 14:33:58.102883375 (503h4m57.97396642s) [noise=true , address=64.49.13.169:4711 , piece_id=N7F5HCN5E32LOF4WTQSZUIT7Z6VACAEYNDWVFHIYFSBKRIT3QXVA, seg_pos={0 0}, upload=441482]: Long tail cancellation due to optimal threshold
Dec 26 14:33:58.102975630 (503h4m57.974058665s) [seg_pos={0 0}, upload=441482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 14:33:58.103068007 (503h4m57.974151052s) [upload=441482]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:33:58.103073868 (503h4m57.974156913s) [upload=441482]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:33:58.391141131 (503h4m58.262224176s) [upload=441482]: Done uploading segments...
Dec 26 14:33:58.391162591 (503h4m58.262245626s) [upload=441482]: Upload finished. err: <nil>
Duration: 48.257579988s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
55 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX.XXXXX.XXXXXXX.XXXXXXXXX.XXXX] duration:
5.001560566s trace:
https://jaeger.dev.storj.io/trace/702499373b7dc423
FAIL (2025-12-26 14:35:26 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001587948s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
14.743539685s trace:
https://jaeger.dev.storj.io/trace/5b63846e68ea1b18
FAIL (2025-12-26 14:35:17 UTC)
orbiter warning: context deadline exceeded
Duration: 14.743575972s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [XX...X...X...........X..X.X...........X....XX.XXX..XXX.X....] duration:
5.764146769s trace:
https://jaeger.dev.storj.io/trace/4957367a1f492fcb
FAIL (2025-12-26 14:35:13 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 14:31:33.597437556 (503h2m33.468520601s) [upload=441449]: Starting upload
Dec 26 14:31:33.597553948 (503h2m33.468636993s) [upload=441449]: Done starting upload
Dec 26 14:31:33.597603222 (503h2m33.468686267s) [upload=441449]: Uploading segments...
Dec 26 14:31:33.597789576 (503h2m33.468872611s) [upload=441449]: Got next segment. Inline: false
Dec 26 14:31:33.597882563 (503h2m33.468965598s) [upload=441449]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 14:31:33.597889186 (503h2m33.468972231s) [upload=441449]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 14:31:33.846933082 (503h2m33.718016127s) [seg_pos={0 0}, upload=441449]: Begin upload segment...
Dec 26 14:31:33.846983659 (503h2m33.718066694s) [seg_pos={0 0}, upload=441449]: Done begin upload segment.
Dec 26 14:31:33.847017223 (503h2m33.718100268s) [noise=true , address=145.79.227.51:4409 , piece_id=DL3T4T3DOTHIPXAZ5BEEM42OCD3FKTU5XOHVWNDIUQE3ERZFLWZQ, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.847050697 (503h2m33.718133732s) [noise=true , address=77.90.31.14:4661 , piece_id=YFYXETEUCFVK7MBM3QCVVSE4FR4SHJBHRZLD4WHTI64RDSGMGQNQ, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.847570256 (503h2m33.718653311s) [noise=true , address=77.90.31.17:4773 , piece_id=FHH435DDQIZKXMHG76N47EK7JD5XC3IOXFUSSFPDFMKBOLBN2OKA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.848112698 (503h2m33.719195744s) [noise=true , address=45.91.116.4:4940 , piece_id=5X6DK3LI6ASO6EYUS6UTWNZDOR7U3IMVBBY6M73IEAZ52RWS2HYA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.848499495 (503h2m33.71958254s) [noise=true , address=145.79.227.49:4337 , piece_id=JVMK4NXGKUWEP2XAAZZWTSE5TEPFYVGKE3FFWDMOVHRK6Z27K4RA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.848875521 (503h2m33.719958556s) [noise=true , address=145.79.227.45:4229 , piece_id=5RKLGUD2ZO5JEAEK3GVEKTKBHYDVOQ5BK5NYJZQUB7P2I5NBFDBA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.849364823 (503h2m33.720447868s) [noise=true , address=145.79.227.21:4473 , piece_id=UCQK72VUEHSCCJBIEZJ4UD7SMFZUPXK4KM6VGB47RPMOOD2HVTHQ, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.849763733 (503h2m33.720846768s) [noise=true , address=45.91.116.14:4190 , piece_id=ZGP23KORYXHV2ZNPQSDTLCR7PR4HCYPVJ7QCTA6DBYHRSNJAGXPA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.850922750 (503h2m33.722005795s) [noise=true , address=145.79.227.36:4518 , piece_id=LJXTQWBSIVHP5JDCT6PEWOICJFW72MGSRPBOYDIYCJEKRRMLGOLA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.851242398 (503h2m33.722325443s) [noise=true , address=45.91.116.5:4298 , piece_id=6AY2UYXYETFVFKWHNY7OHUONOYAMA26N2YF3RHAYFA2ZZW2ZNQ3Q, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.852200212 (503h2m33.723283247s) [noise=true , address=145.79.227.47:4266 , piece_id=VRNGVBKFWQRJA7TAR6QZ52JVFLKTCNBOYK7VSZSUUJVPFAJBA6GA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.853435684 (503h2m33.72451873s) [noise=true , address=145.79.227.38:4979 , piece_id=52FH5WDZ76TN3LRN6IMMTS5OE3TXC637YY53IAEU7MHGHRE7R7JA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.854620921 (503h2m33.725703956s) [noise=true , address=145.79.227.50:4375 , piece_id=23RC4QNCFLTCIF4JHVD7ESHDIDW2CWWKCSE5Z64YX3P5526GDCJA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.856049742 (503h2m33.727132787s) [noise=true , address=77.90.31.19:4844 , piece_id=O56QQ4AUPE24ERTZQYUQEOK6NYNWYI3DNZQ5FWKFEZLLHF63OANQ, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.857977794 (503h2m33.729060839s) [noise=true , address=77.90.31.11:4555 , piece_id=W7NU2VVWGDDND474EICGZYS3GPOR75FNBD7Q6ZVIZVEQ2CRYCNKQ, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.858982477 (503h2m33.730065512s) [noise=true , address=145.79.227.29:4943 , piece_id=Y7JPU43H23ZITBIQ6TK7RUZPG3T3NMGPDTB7S6DJNZG7IVGTUYVA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:35.571477638 (503h2m35.442560673s) [upload=441449]: Next returned nil segment
Dec 26 14:31:35.571490583 (503h2m35.442573628s) [upload=441449]: Waiting for error group managing segments...
Dec 26 14:31:35.855026257 (503h2m35.726109302s) [noise=true , address=145.79.227.49:4337 , piece_id=JVMK4NXGKUWEP2XAAZZWTSE5TEPFYVGKE3FFWDMOVHRK6Z27K4RA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:35.862095689 (503h2m35.733178734s) [noise=true , address=145.79.227.38:4979 , piece_id=52FH5WDZ76TN3LRN6IMMTS5OE3TXC637YY53IAEU7MHGHRE7R7JA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:35.862796403 (503h2m35.733879438s) [noise=true , address=145.79.227.36:4518 , piece_id=LJXTQWBSIVHP5JDCT6PEWOICJFW72MGSRPBOYDIYCJEKRRMLGOLA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:35.864289396 (503h2m35.735372431s) [noise=true , address=145.79.227.50:4375 , piece_id=23RC4QNCFLTCIF4JHVD7ESHDIDW2CWWKCSE5Z64YX3P5526GDCJA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:35.869609206 (503h2m35.740692251s) [noise=true , address=145.79.227.45:4229 , piece_id=5RKLGUD2ZO5JEAEK3GVEKTKBHYDVOQ5BK5NYJZQUB7P2I5NBFDBA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:35.872899100 (503h2m35.743982135s) [noise=true , address=145.79.227.21:4473 , piece_id=UCQK72VUEHSCCJBIEZJ4UD7SMFZUPXK4KM6VGB47RPMOOD2HVTHQ, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:35.881408674 (503h2m35.752491709s) [noise=true , address=145.79.227.29:4943 , piece_id=Y7JPU43H23ZITBIQ6TK7RUZPG3T3NMGPDTB7S6DJNZG7IVGTUYVA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:35.888713384 (503h2m35.759796429s) [noise=true , address=145.79.227.51:4409 , piece_id=DL3T4T3DOTHIPXAZ5BEEM42OCD3FKTU5XOHVWNDIUQE3ERZFLWZQ, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:35.893099275 (503h2m35.76418231s) [noise=true , address=145.79.227.47:4266 , piece_id=VRNGVBKFWQRJA7TAR6QZ52JVFLKTCNBOYK7VSZSUUJVPFAJBA6GA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:36.043138096 (503h2m35.914221141s) [noise=true , address=77.90.31.14:4661 , piece_id=YFYXETEUCFVK7MBM3QCVVSE4FR4SHJBHRZLD4WHTI64RDSGMGQNQ, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:36.087256205 (503h2m35.95833924s) [noise=true , address=45.91.116.5:4298 , piece_id=6AY2UYXYETFVFKWHNY7OHUONOYAMA26N2YF3RHAYFA2ZZW2ZNQ3Q, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:36.233410791 (503h2m36.104493836s) [noise=true , address=45.91.116.14:4190 , piece_id=ZGP23KORYXHV2ZNPQSDTLCR7PR4HCYPVJ7QCTA6DBYHRSNJAGXPA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:36.993343372 (503h2m36.864426407s) [noise=true , address=77.90.31.19:4844 , piece_id=O56QQ4AUPE24ERTZQYUQEOK6NYNWYI3DNZQ5FWKFEZLLHF63OANQ, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:36.993368200 (503h2m36.864451285s) [seg_pos={0 0}, upload=441449]: Segment reached optimal threshold of 13 pieces.
Dec 26 14:31:36.993595764 (503h2m36.864678799s) [noise=true , address=77.90.31.11:4555 , piece_id=W7NU2VVWGDDND474EICGZYS3GPOR75FNBD7Q6ZVIZVEQ2CRYCNKQ, seg_pos={0 0}, upload=441449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: write tcp 188.166.225.170:46760->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:46760->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46760->77.90.31.11:4555: use of closed network connection
Dec 26 14:31:36.993605442 (503h2m36.864688477s) [noise=true , address=77.90.31.11:4555 , piece_id=W7NU2VVWGDDND474EICGZYS3GPOR75FNBD7Q6ZVIZVEQ2CRYCNKQ, seg_pos={0 0}, upload=441449]: Long tail cancellation due to optimal threshold
Dec 26 14:31:36.993751880 (503h2m36.864834946s) [noise=true , address=77.90.31.17:4773 , piece_id=FHH435DDQIZKXMHG76N47EK7JD5XC3IOXFUSSFPDFMKBOLBN2OKA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF
Dec 26 14:31:36.993770246 (503h2m36.864853291s) [noise=true , address=77.90.31.17:4773 , piece_id=FHH435DDQIZKXMHG76N47EK7JD5XC3IOXFUSSFPDFMKBOLBN2OKA, seg_pos={0 0}, upload=441449]: Long tail cancellation due to optimal threshold
Dec 26 14:31:36.993934509 (503h2m36.865017544s) [noise=true , address=45.91.116.4:4940 , piece_id=5X6DK3LI6ASO6EYUS6UTWNZDOR7U3IMVBBY6M73IEAZ52RWS2HYA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF
Dec 26 14:31:36.993942774 (503h2m36.865025809s) [noise=true , address=45.91.116.4:4940 , piece_id=5X6DK3LI6ASO6EYUS6UTWNZDOR7U3IMVBBY6M73IEAZ52RWS2HYA, seg_pos={0 0}, upload=441449]: Long tail cancellation due to optimal threshold
Dec 26 14:31:36.993961339 (503h2m36.865044384s) [seg_pos={0 0}, upload=441449]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 14:31:36.994038988 (503h2m36.865122033s) [upload=441449]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:31:36.994045901 (503h2m36.865128966s) [upload=441449]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:31:37.298152335 (503h2m37.16923537s) [upload=441449]: Done uploading segments...
Dec 26 14:31:37.298171903 (503h2m37.169254938s) [upload=441449]: Upload finished. err: <nil>
Duration: 5.764174491s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
9.726068562s trace:
https://jaeger.dev.storj.io/trace/3ccdb98964ad100c
FAIL (2025-12-26 14:35:06 UTC)
orbiter warning: context deadline exceeded
Duration: 9.726097546s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
19 out of 60 is failed (threshold=3) [X.....X..........X..X..X.X.X....X.....XXX.XX.XX.....X...XX.X] duration:
1m0.003209148s trace:
https://jaeger.dev.storj.io/trace/7e11b9275a734af0
FAIL (2025-12-26 14:34:54 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
26 14:33:55.235683223 (503h4m55.106766268s) [noise=true , address=77.90.31.13:4650 , piece_id=LNQNQRIHW4TRCOEZFGANMG7IWB5O6MHVTS4BL6UTI34AL5OIOLKA, seg_pos={0 0}, upload=441482]: Uploading piece...
Dec 26 14:33:55.236058588 (503h4m55.107141633s) [noise=true , address=145.79.227.30:4780 , piece_id=NDJPRC7S4BLPP757YCO3RPHXVPPBP7333K4NRPE6GBDBKTC5XB6A, seg_pos={0 0}, upload=441482]: Uploading piece...
Dec 26 14:33:55.236348910 (503h4m55.107431955s) [noise=true , address=45.91.116.13:4160 , piece_id=BHHK2DRNWQNJEYF5APSN6RPBSRC6GLOAVDMHC3QAFE3RKGTWMFNQ, seg_pos={0 0}, upload=441482]: Uploading piece...
Dec 26 14:33:55.236652318 (503h4m55.107735353s) [noise=true , address=145.79.227.44:4197 , piece_id=SAKARE65PJ3VL56Z272CJLUQ3XOSHSV4VHPZTCCOLEGUXBKYUGMQ, seg_pos={0 0}, upload=441482]: Uploading piece...
Dec 26 14:33:55.236937321 (503h4m55.108020356s) [noise=true , address=77.90.31.16:4743 , piece_id=USSS6LMMUVCNGYVZMBCXC72OH5SPJKF64P5CVT65BIFWCRPC6NAQ, seg_pos={0 0}, upload=441482]: Uploading piece...
Dec 26 14:33:55.237356940 (503h4m55.108439985s) [noise=true , address=64.49.13.138:4489 , piece_id=FYQKAISGAQRIXFB2IIUIPYOY5NDF5YEHWSFZ5FPS7WQDXFZ6BWFQ, seg_pos={0 0}, upload=441482]: Uploading piece...
Dec 26 14:33:55.237659506 (503h4m55.108742551s) [noise=true , address=145.79.227.40:4059 , piece_id=XLYE4JHTS56AKAG2SI7RWMDC6IGROOFNQA6NO6EZG6DZADGN65DQ, seg_pos={0 0}, upload=441482]: Uploading piece...
Dec 26 14:33:55.237988353 (503h4m55.109071398s) [noise=true , address=77.90.31.17:4769 , piece_id=WH7Z3ZN67MRXPVTDIVT57CRPMUXVVQMSTFVICQB5M3MEBV7LALTQ, seg_pos={0 0}, upload=441482]: Uploading piece...
Dec 26 14:33:55.238279647 (503h4m55.109362682s) [noise=true , address=77.90.31.21:4903 , piece_id=F5WG3S4YBD7NQ2VCA3HWVOYBRRHPEYRITFTEKPHINL7X6Q5UGY6A, seg_pos={0 0}, upload=441482]: Uploading piece...
Dec 26 14:33:55.238702262 (503h4m55.109785307s) [noise=true , address=77.90.31.14:4679 , piece_id=IXEPYMDUICJM6YAPX5DDQETYZQ6ZJG43FTAGYKX2MAX2G2QWGVLA, seg_pos={0 0}, upload=441482]: Uploading piece...
Dec 26 14:33:55.239070814 (503h4m55.110153849s) [noise=true , address=145.79.227.52:4469 , piece_id=EOIJ6Z4OKARUW3KNGGNEYBQ2QRLRSGEJTPZ7TZBRV4NBAJHAQ6GQ, seg_pos={0 0}, upload=441482]: Uploading piece...
Dec 26 14:33:55.240243927 (503h4m55.111326972s) [noise=true , address=77.90.31.12:4607 , piece_id=HFZCUOPCYLCU6PQGH2UHKIHIVLORJG4MTM4CCWA7XJ6H6JBAPFYQ, seg_pos={0 0}, upload=441482]: Uploading piece...
Dec 26 14:33:55.240758477 (503h4m55.111841512s) [noise=true , address=77.90.31.15:4704 , piece_id=IW2IDZMLW3WLTTUIVGJMB4CTIL7U5INVUNYP7AV4QBERU6J6NNCA, seg_pos={0 0}, upload=441482]: Uploading piece...
Dec 26 14:33:55.242161258 (503h4m55.113244293s) [noise=true , address=77.90.31.11:4564 , piece_id=GOJICK6HKWQKBNICAUWE7KWHZCPSTVRUTNQOHVYK2I6B7Z46PS7A, seg_pos={0 0}, upload=441482]: Uploading piece...
Dec 26 14:33:55.243174468 (503h4m55.114257513s) [noise=true , address=64.49.13.155:4920 , piece_id=RM6ZUHHBVOFC7REQZ5A2IOJK2ZNOL2XYFIDXQIBSJBKPHW3NJSLQ, seg_pos={0 0}, upload=441482]: Uploading piece...
Dec 26 14:33:55.244432463 (503h4m55.115515509s) [noise=true , address=77.90.31.18:4834 , piece_id=6ONKW2LMMA5HVDGAW2ZY5KIU5BOL6PRK6F2XQ6DZJMIG2B4ZB2ZA, seg_pos={0 0}, upload=441482]: Uploading piece...
Dec 26 14:33:56.587466035 (503h4m56.45854908s) [upload=441482]: Next returned nil segment
Dec 26 14:33:56.587504639 (503h4m56.458587674s) [upload=441482]: Waiting for error group managing segments...
Dec 26 14:33:56.941402434 (503h4m56.812485479s) [noise=true , address=64.49.13.155:4920 , piece_id=RM6ZUHHBVOFC7REQZ5A2IOJK2ZNOL2XYFIDXQIBSJBKPHW3NJSLQ, seg_pos={0 0}, upload=441482]: Done uploading piece. err: <nil>
Dec 26 14:33:57.105592201 (503h4m56.976675236s) [noise=true , address=145.79.227.40:4059 , piece_id=XLYE4JHTS56AKAG2SI7RWMDC6IGROOFNQA6NO6EZG6DZADGN65DQ, seg_pos={0 0}, upload=441482]: Done uploading piece. err: <nil>
Dec 26 14:33:57.109300512 (503h4m56.980383548s) [noise=true , address=145.79.227.52:4469 , piece_id=EOIJ6Z4OKARUW3KNGGNEYBQ2QRLRSGEJTPZ7TZBRV4NBAJHAQ6GQ, seg_pos={0 0}, upload=441482]: Done uploading piece. err: <nil>
Dec 26 14:33:57.310829930 (503h4m57.181912965s) [noise=true , address=77.90.31.15:4704 , piece_id=IW2IDZMLW3WLTTUIVGJMB4CTIL7U5INVUNYP7AV4QBERU6J6NNCA, seg_pos={0 0}, upload=441482]: Done uploading piece. err: <nil>
Dec 26 14:33:57.325869969 (503h4m57.196953034s) [noise=true , address=77.90.31.18:4834 , piece_id=6ONKW2LMMA5HVDGAW2ZY5KIU5BOL6PRK6F2XQ6DZJMIG2B4ZB2ZA, seg_pos={0 0}, upload=441482]: Done uploading piece. err: <nil>
Dec 26 14:33:57.434979604 (503h4m57.306062639s) [noise=true , address=77.90.31.17:4769 , piece_id=WH7Z3ZN67MRXPVTDIVT57CRPMUXVVQMSTFVICQB5M3MEBV7LALTQ, seg_pos={0 0}, upload=441482]: Done uploading piece. err: <nil>
Dec 26 14:33:57.498015707 (503h4m57.369098742s) [noise=true , address=45.91.116.13:4160 , piece_id=BHHK2DRNWQNJEYF5APSN6RPBSRC6GLOAVDMHC3QAFE3RKGTWMFNQ, seg_pos={0 0}, upload=441482]: Done uploading piece. err: <nil>
Dec 26 14:33:57.795622655 (503h4m57.6667057s) [noise=true , address=77.90.31.13:4650 , piece_id=LNQNQRIHW4TRCOEZFGANMG7IWB5O6MHVTS4BL6UTI34AL5OIOLKA, seg_pos={0 0}, upload=441482]: Done uploading piece. err: <nil>
Dec 26 14:33:57.958552623 (503h4m57.829635658s) [noise=true , address=77.90.31.11:4564 , piece_id=GOJICK6HKWQKBNICAUWE7KWHZCPSTVRUTNQOHVYK2I6B7Z46PS7A, seg_pos={0 0}, upload=441482]: Done uploading piece. err: <nil>
Dec 26 14:33:57.984892960 (503h4m57.855976005s) [noise=true , address=77.90.31.21:4903 , piece_id=F5WG3S4YBD7NQ2VCA3HWVOYBRRHPEYRITFTEKPHINL7X6Q5UGY6A, seg_pos={0 0}, upload=441482]: Done uploading piece. err: <nil>
Dec 26 14:33:58.006177078 (503h4m57.877260113s) [noise=true , address=145.79.227.44:4197 , piece_id=SAKARE65PJ3VL56Z272CJLUQ3XOSHSV4VHPZTCCOLEGUXBKYUGMQ, seg_pos={0 0}, upload=441482]: Done uploading piece. err: <nil>
Dec 26 14:33:58.040690618 (503h4m57.911773653s) [noise=true , address=45.91.116.10:4114 , piece_id=63CCS6OE5TP7RI6QMO3VPGE5JI4T3EOXHI5ZR5ZGB3I75WECUYSQ, seg_pos={0 0}, upload=441482]: Done uploading piece. err: <nil>
Dec 26 14:33:58.063917928 (503h4m57.935000963s) [noise=true , address=145.79.227.30:4780 , piece_id=NDJPRC7S4BLPP757YCO3RPHXVPPBP7333K4NRPE6GBDBKTC5XB6A, seg_pos={0 0}, upload=441482]: Done uploading piece. err: <nil>
Dec 26 14:33:58.102298952 (503h4m57.973381997s) [noise=true , address=77.90.31.16:4743 , piece_id=USSS6LMMUVCNGYVZMBCXC72OH5SPJKF64P5CVT65BIFWCRPC6NAQ, seg_pos={0 0}, upload=441482]: Done uploading piece. err: <nil>
Dec 26 14:33:58.102329440 (503h4m57.973412476s) [seg_pos={0 0}, upload=441482]: Segment reached optimal threshold of 14 pieces.
Dec 26 14:33:58.102543318 (503h4m57.973626363s) [noise=true , address=77.90.31.14:4679 , piece_id=IXEPYMDUICJM6YAPX5DDQETYZQ6ZJG43FTAGYKX2MAX2G2QWGVLA, seg_pos={0 0}, upload=441482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k29MGpez3ByMHnG8qGBm9iPwo3VrKUGDHpz9CqGEGNn47rxDx): protocol: expected piece hash; context canceled; EOF
Dec 26 14:33:58.102550972 (503h4m57.973634007s) [noise=true , address=77.90.31.14:4679 , piece_id=IXEPYMDUICJM6YAPX5DDQETYZQ6ZJG43FTAGYKX2MAX2G2QWGVLA, seg_pos={0 0}, upload=441482]: Long tail cancellation due to optimal threshold
Dec 26 14:33:58.102670821 (503h4m57.973753876s) [noise=true , address=77.90.31.12:4607 , piece_id=HFZCUOPCYLCU6PQGH2UHKIHIVLORJG4MTM4CCWA7XJ6H6JBAPFYQ, seg_pos={0 0}, upload=441482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm): protocol: expected piece hash; context canceled; EOF
Dec 26 14:33:58.102680829 (503h4m57.973763875s) [noise=true , address=77.90.31.12:4607 , piece_id=HFZCUOPCYLCU6PQGH2UHKIHIVLORJG4MTM4CCWA7XJ6H6JBAPFYQ, seg_pos={0 0}, upload=441482]: Long tail cancellation due to optimal threshold
Dec 26 14:33:58.102713862 (503h4m57.973796908s) [noise=true , address=64.49.13.138:4489 , piece_id=FYQKAISGAQRIXFB2IIUIPYOY5NDF5YEHWSFZ5FPS7WQDXFZ6BWFQ, seg_pos={0 0}, upload=441482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CiiMtdAoMhPZwSqwraaQcrrx4vZw6L74vadw712VQbRLpNzKN): protocol: expected piece hash; context canceled; EOF
Dec 26 14:33:58.102720966 (503h4m57.973804011s) [noise=true , address=64.49.13.138:4489 , piece_id=FYQKAISGAQRIXFB2IIUIPYOY5NDF5YEHWSFZ5FPS7WQDXFZ6BWFQ, seg_pos={0 0}, upload=441482]: Long tail cancellation due to optimal threshold
Dec 26 14:33:58.102875280 (503h4m57.973958335s) [noise=true , address=64.49.13.169:4711 , piece_id=N7F5HCN5E32LOF4WTQSZUIT7Z6VACAEYNDWVFHIYFSBKRIT3QXVA, seg_pos={0 0}, upload=441482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:56296->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:56296->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56296->64.49.13.169:4711: use of closed network connection
Dec 26 14:33:58.102883375 (503h4m57.97396642s) [noise=true , address=64.49.13.169:4711 , piece_id=N7F5HCN5E32LOF4WTQSZUIT7Z6VACAEYNDWVFHIYFSBKRIT3QXVA, seg_pos={0 0}, upload=441482]: Long tail cancellation due to optimal threshold
Dec 26 14:33:58.102975630 (503h4m57.974058665s) [seg_pos={0 0}, upload=441482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 14:33:58.103068007 (503h4m57.974151052s) [upload=441482]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:33:58.103073868 (503h4m57.974156913s) [upload=441482]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:33:58.391141131 (503h4m58.262224176s) [upload=441482]: Done uploading segments...
Dec 26 14:33:58.391162591 (503h4m58.262245626s) [upload=441482]: Upload finished. err: <nil>
Duration: 1m0.003224267s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
11.336012479s trace:
https://jaeger.dev.storj.io/trace/5b63846e68ea1b18
FAIL (2025-12-26 14:34:52 UTC)
orbiter warning: context deadline exceeded
Duration: 11.33602839s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
13.974641472s trace:
https://jaeger.dev.storj.io/trace/3ccdb98964ad100c
FAIL (2025-12-26 14:34:48 UTC)
orbiter warning: context deadline exceeded
Duration: 13.974662832s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
55 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX.XXXXX.XXXXXXX.XXXXXXXXX.XXX] duration:
5.000679036s trace:
https://jaeger.dev.storj.io/trace/66b6fbf2a0ff4ce1
FAIL (2025-12-26 14:34:29 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000700688s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [.XX...X...X...........X..X.X...........X....XX.XXX..XXX.X...] duration:
5.792715329s trace:
https://jaeger.dev.storj.io/trace/4957367a1f492fcb
FAIL (2025-12-26 14:34:08 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 14:31:33.597437556 (503h2m33.468520601s) [upload=441449]: Starting upload
Dec 26 14:31:33.597553948 (503h2m33.468636993s) [upload=441449]: Done starting upload
Dec 26 14:31:33.597603222 (503h2m33.468686267s) [upload=441449]: Uploading segments...
Dec 26 14:31:33.597789576 (503h2m33.468872611s) [upload=441449]: Got next segment. Inline: false
Dec 26 14:31:33.597882563 (503h2m33.468965598s) [upload=441449]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 14:31:33.597889186 (503h2m33.468972231s) [upload=441449]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 14:31:33.846933082 (503h2m33.718016127s) [seg_pos={0 0}, upload=441449]: Begin upload segment...
Dec 26 14:31:33.846983659 (503h2m33.718066694s) [seg_pos={0 0}, upload=441449]: Done begin upload segment.
Dec 26 14:31:33.847017223 (503h2m33.718100268s) [noise=true , address=145.79.227.51:4409 , piece_id=DL3T4T3DOTHIPXAZ5BEEM42OCD3FKTU5XOHVWNDIUQE3ERZFLWZQ, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.847050697 (503h2m33.718133732s) [noise=true , address=77.90.31.14:4661 , piece_id=YFYXETEUCFVK7MBM3QCVVSE4FR4SHJBHRZLD4WHTI64RDSGMGQNQ, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.847570256 (503h2m33.718653311s) [noise=true , address=77.90.31.17:4773 , piece_id=FHH435DDQIZKXMHG76N47EK7JD5XC3IOXFUSSFPDFMKBOLBN2OKA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.848112698 (503h2m33.719195744s) [noise=true , address=45.91.116.4:4940 , piece_id=5X6DK3LI6ASO6EYUS6UTWNZDOR7U3IMVBBY6M73IEAZ52RWS2HYA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.848499495 (503h2m33.71958254s) [noise=true , address=145.79.227.49:4337 , piece_id=JVMK4NXGKUWEP2XAAZZWTSE5TEPFYVGKE3FFWDMOVHRK6Z27K4RA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.848875521 (503h2m33.719958556s) [noise=true , address=145.79.227.45:4229 , piece_id=5RKLGUD2ZO5JEAEK3GVEKTKBHYDVOQ5BK5NYJZQUB7P2I5NBFDBA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.849364823 (503h2m33.720447868s) [noise=true , address=145.79.227.21:4473 , piece_id=UCQK72VUEHSCCJBIEZJ4UD7SMFZUPXK4KM6VGB47RPMOOD2HVTHQ, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.849763733 (503h2m33.720846768s) [noise=true , address=45.91.116.14:4190 , piece_id=ZGP23KORYXHV2ZNPQSDTLCR7PR4HCYPVJ7QCTA6DBYHRSNJAGXPA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.850922750 (503h2m33.722005795s) [noise=true , address=145.79.227.36:4518 , piece_id=LJXTQWBSIVHP5JDCT6PEWOICJFW72MGSRPBOYDIYCJEKRRMLGOLA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.851242398 (503h2m33.722325443s) [noise=true , address=45.91.116.5:4298 , piece_id=6AY2UYXYETFVFKWHNY7OHUONOYAMA26N2YF3RHAYFA2ZZW2ZNQ3Q, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.852200212 (503h2m33.723283247s) [noise=true , address=145.79.227.47:4266 , piece_id=VRNGVBKFWQRJA7TAR6QZ52JVFLKTCNBOYK7VSZSUUJVPFAJBA6GA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.853435684 (503h2m33.72451873s) [noise=true , address=145.79.227.38:4979 , piece_id=52FH5WDZ76TN3LRN6IMMTS5OE3TXC637YY53IAEU7MHGHRE7R7JA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.854620921 (503h2m33.725703956s) [noise=true , address=145.79.227.50:4375 , piece_id=23RC4QNCFLTCIF4JHVD7ESHDIDW2CWWKCSE5Z64YX3P5526GDCJA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.856049742 (503h2m33.727132787s) [noise=true , address=77.90.31.19:4844 , piece_id=O56QQ4AUPE24ERTZQYUQEOK6NYNWYI3DNZQ5FWKFEZLLHF63OANQ, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.857977794 (503h2m33.729060839s) [noise=true , address=77.90.31.11:4555 , piece_id=W7NU2VVWGDDND474EICGZYS3GPOR75FNBD7Q6ZVIZVEQ2CRYCNKQ, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.858982477 (503h2m33.730065512s) [noise=true , address=145.79.227.29:4943 , piece_id=Y7JPU43H23ZITBIQ6TK7RUZPG3T3NMGPDTB7S6DJNZG7IVGTUYVA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:35.571477638 (503h2m35.442560673s) [upload=441449]: Next returned nil segment
Dec 26 14:31:35.571490583 (503h2m35.442573628s) [upload=441449]: Waiting for error group managing segments...
Dec 26 14:31:35.855026257 (503h2m35.726109302s) [noise=true , address=145.79.227.49:4337 , piece_id=JVMK4NXGKUWEP2XAAZZWTSE5TEPFYVGKE3FFWDMOVHRK6Z27K4RA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:35.862095689 (503h2m35.733178734s) [noise=true , address=145.79.227.38:4979 , piece_id=52FH5WDZ76TN3LRN6IMMTS5OE3TXC637YY53IAEU7MHGHRE7R7JA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:35.862796403 (503h2m35.733879438s) [noise=true , address=145.79.227.36:4518 , piece_id=LJXTQWBSIVHP5JDCT6PEWOICJFW72MGSRPBOYDIYCJEKRRMLGOLA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:35.864289396 (503h2m35.735372431s) [noise=true , address=145.79.227.50:4375 , piece_id=23RC4QNCFLTCIF4JHVD7ESHDIDW2CWWKCSE5Z64YX3P5526GDCJA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:35.869609206 (503h2m35.740692251s) [noise=true , address=145.79.227.45:4229 , piece_id=5RKLGUD2ZO5JEAEK3GVEKTKBHYDVOQ5BK5NYJZQUB7P2I5NBFDBA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:35.872899100 (503h2m35.743982135s) [noise=true , address=145.79.227.21:4473 , piece_id=UCQK72VUEHSCCJBIEZJ4UD7SMFZUPXK4KM6VGB47RPMOOD2HVTHQ, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:35.881408674 (503h2m35.752491709s) [noise=true , address=145.79.227.29:4943 , piece_id=Y7JPU43H23ZITBIQ6TK7RUZPG3T3NMGPDTB7S6DJNZG7IVGTUYVA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:35.888713384 (503h2m35.759796429s) [noise=true , address=145.79.227.51:4409 , piece_id=DL3T4T3DOTHIPXAZ5BEEM42OCD3FKTU5XOHVWNDIUQE3ERZFLWZQ, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:35.893099275 (503h2m35.76418231s) [noise=true , address=145.79.227.47:4266 , piece_id=VRNGVBKFWQRJA7TAR6QZ52JVFLKTCNBOYK7VSZSUUJVPFAJBA6GA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:36.043138096 (503h2m35.914221141s) [noise=true , address=77.90.31.14:4661 , piece_id=YFYXETEUCFVK7MBM3QCVVSE4FR4SHJBHRZLD4WHTI64RDSGMGQNQ, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:36.087256205 (503h2m35.95833924s) [noise=true , address=45.91.116.5:4298 , piece_id=6AY2UYXYETFVFKWHNY7OHUONOYAMA26N2YF3RHAYFA2ZZW2ZNQ3Q, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:36.233410791 (503h2m36.104493836s) [noise=true , address=45.91.116.14:4190 , piece_id=ZGP23KORYXHV2ZNPQSDTLCR7PR4HCYPVJ7QCTA6DBYHRSNJAGXPA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:36.993343372 (503h2m36.864426407s) [noise=true , address=77.90.31.19:4844 , piece_id=O56QQ4AUPE24ERTZQYUQEOK6NYNWYI3DNZQ5FWKFEZLLHF63OANQ, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:36.993368200 (503h2m36.864451285s) [seg_pos={0 0}, upload=441449]: Segment reached optimal threshold of 13 pieces.
Dec 26 14:31:36.993595764 (503h2m36.864678799s) [noise=true , address=77.90.31.11:4555 , piece_id=W7NU2VVWGDDND474EICGZYS3GPOR75FNBD7Q6ZVIZVEQ2CRYCNKQ, seg_pos={0 0}, upload=441449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: write tcp 188.166.225.170:46760->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:46760->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46760->77.90.31.11:4555: use of closed network connection
Dec 26 14:31:36.993605442 (503h2m36.864688477s) [noise=true , address=77.90.31.11:4555 , piece_id=W7NU2VVWGDDND474EICGZYS3GPOR75FNBD7Q6ZVIZVEQ2CRYCNKQ, seg_pos={0 0}, upload=441449]: Long tail cancellation due to optimal threshold
Dec 26 14:31:36.993751880 (503h2m36.864834946s) [noise=true , address=77.90.31.17:4773 , piece_id=FHH435DDQIZKXMHG76N47EK7JD5XC3IOXFUSSFPDFMKBOLBN2OKA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF
Dec 26 14:31:36.993770246 (503h2m36.864853291s) [noise=true , address=77.90.31.17:4773 , piece_id=FHH435DDQIZKXMHG76N47EK7JD5XC3IOXFUSSFPDFMKBOLBN2OKA, seg_pos={0 0}, upload=441449]: Long tail cancellation due to optimal threshold
Dec 26 14:31:36.993934509 (503h2m36.865017544s) [noise=true , address=45.91.116.4:4940 , piece_id=5X6DK3LI6ASO6EYUS6UTWNZDOR7U3IMVBBY6M73IEAZ52RWS2HYA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF
Dec 26 14:31:36.993942774 (503h2m36.865025809s) [noise=true , address=45.91.116.4:4940 , piece_id=5X6DK3LI6ASO6EYUS6UTWNZDOR7U3IMVBBY6M73IEAZ52RWS2HYA, seg_pos={0 0}, upload=441449]: Long tail cancellation due to optimal threshold
Dec 26 14:31:36.993961339 (503h2m36.865044384s) [seg_pos={0 0}, upload=441449]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 14:31:36.994038988 (503h2m36.865122033s) [upload=441449]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:31:36.994045901 (503h2m36.865128966s) [upload=441449]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:31:37.298152335 (503h2m37.16923537s) [upload=441449]: Done uploading segments...
Dec 26 14:31:37.298171903 (503h2m37.169254938s) [upload=441449]: Upload finished. err: <nil>
Duration: 5.792753812s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
19 out of 60 is failed (threshold=3) [XX.....X..........X..X..X.X.X....X.....XXX.XX.XX.....X...XX.] duration:
48.395864333s trace:
https://jaeger.dev.storj.io/trace/6da2926055e33842
FAIL (2025-12-26 14:33:54 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12qXhyMbfXELW1Vg5z9NpqFX1fm3v2pGEaA56qSc8kHefPNNNYs, Piece ID: A2EPWATZFDQ4S3UV6SKFERNXSAMFNQ7VRUAUKRNHJWGDYJ7Q6FBQ) context deadline exceeded; piecestore: (Node ID: 12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D, Piece ID: EPGIESRMH2ZMNMOEPKV3WLRHZ7DMHSDYP45GTOCD4VV4NS4ZQ6PQ) context deadline exceeded; piecestore: (Node ID: 1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY, Piece ID: ZEGVK3MHJK364MPDTSQDNNKXPJRBUIZMPRNPYBYKSEDMOGPE2KTA) context deadline exceeded; piecestore: (Node ID: 1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, Piece ID: XBQP5JFWRJGF3R3HBTBJSWHTU6TDXFTZZRLO2DCPBD6LTBIVCXGA) context deadline exceeded; piecestore: (Node ID: 124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, Piece ID: JIERBYN26O3X36AUJLU4OUHY22GQE3YWQUOTQ5N3KNGQVAR4O27A) context deadline exceeded
======LOGS======
piece...
Dec 26 14:32:06.384298372 (503h3m6.255381407s) [noise=true , address=77.90.31.19:4863 , piece_id=JIERBYN26O3X36AUJLU4OUHY22GQE3YWQUOTQ5N3KNGQVAR4O27A, seg_pos={0 0}, upload=441454]: Uploading piece...
Dec 26 14:32:06.384615456 (503h3m6.255698501s) [noise=true , address=77.90.31.11:4580 , piece_id=EPGIESRMH2ZMNMOEPKV3WLRHZ7DMHSDYP45GTOCD4VV4NS4ZQ6PQ, seg_pos={0 0}, upload=441454]: Uploading piece...
Dec 26 14:32:06.384930195 (503h3m6.25601324s) [noise=true , address=77.90.31.15:4711 , piece_id=VY5TQ2WLI2XJZ2RJ2UNT2VA6E7S6U2DYF243MWK7N7AAOJWIDF4A, seg_pos={0 0}, upload=441454]: Uploading piece...
Dec 26 14:32:06.385356466 (503h3m6.256439511s) [noise=true , address=45.91.116.4:4967 , piece_id=AJWSUN2SN4FD3EE3FNISARGF34G5QRW76KBHXQSMI2AJJ33J5FPQ, seg_pos={0 0}, upload=441454]: Uploading piece...
Dec 26 14:32:06.385694320 (503h3m6.256777365s) [noise=true , address=77.90.31.12:4604 , piece_id=XBQP5JFWRJGF3R3HBTBJSWHTU6TDXFTZZRLO2DCPBD6LTBIVCXGA, seg_pos={0 0}, upload=441454]: Uploading piece...
Dec 26 14:32:06.386092868 (503h3m6.257175903s) [noise=true , address=145.79.227.3:4153 , piece_id=PHL7TPZI2LJJF3PC4YSO3UUKI7HH25LL5VSCK3CVVYSE6PN23NTQ, seg_pos={0 0}, upload=441454]: Uploading piece...
Dec 26 14:32:06.387802996 (503h3m6.258886041s) [noise=true , address=145.79.227.40:4068 , piece_id=TBIDPYRQBO7QAUEEGJYLA5YQD5YSIJYWIXHO6KUMFRM2WW3KRCLQ, seg_pos={0 0}, upload=441454]: Uploading piece...
Dec 26 14:32:06.389673156 (503h3m6.260756201s) [noise=true , address=193.149.164.143:4700 , piece_id=ZEGVK3MHJK364MPDTSQDNNKXPJRBUIZMPRNPYBYKSEDMOGPE2KTA, seg_pos={0 0}, upload=441454]: Uploading piece...
Dec 26 14:32:06.391484555 (503h3m6.2625676s) [noise=true , address=64.49.13.169:4716 , piece_id=B3MX3TD6WCWHUSEJAHIUQWBQJJMY7M4ELXZWC6DGK5UG734HZU4A, seg_pos={0 0}, upload=441454]: Uploading piece...
Dec 26 14:32:06.393370396 (503h3m6.264453441s) [noise=true , address=77.90.31.18:4824 , piece_id=DMVU5JF44YYBFO6W57RLOVDSGMK7RCIBGKSRP5WZUWALNZFZHEMA, seg_pos={0 0}, upload=441454]: Uploading piece...
Dec 26 14:32:06.395059202 (503h3m6.266142247s) [noise=true , address=145.79.227.44:4197 , piece_id=3T7KOR4LKB7IQELOZMTZU4STM2MCHGQ7GN4O2PJHY5HQJTH7Q4OA, seg_pos={0 0}, upload=441454]: Uploading piece...
Dec 26 14:32:06.397013243 (503h3m6.268096288s) [noise=true , address=145.79.227.30:4794 , piece_id=2SAB6NZV3SRNENOPYFOAQZIR3TK7CR2XV6YBZYGYQSWW3G4TFV4Q, seg_pos={0 0}, upload=441454]: Uploading piece...
Dec 26 14:32:06.399007981 (503h3m6.270091026s) [noise=true , address=145.79.227.34:4936 , piece_id=IJWHOOUSWAJ3MWMAHIYYJSCK5C5BX4DJM4GDQMQDZPP7TG6GGFTQ, seg_pos={0 0}, upload=441454]: Uploading piece...
Dec 26 14:32:06.400585856 (503h3m6.271668901s) [noise=true , address=145.79.227.48:4329 , piece_id=TYTZI6WQKX7U3F6WS4E5NXHZ7NDCW3EYGB3E7ZDV4YAU5642CJFA, seg_pos={0 0}, upload=441454]: Uploading piece...
Dec 26 14:32:06.402088056 (503h3m6.273171102s) [noise=true , address=77.90.31.14:4695 , piece_id=LX2V6PMVTQPMIVIRRM7FIWYC6FLW2NTCVVRZ4QDSETZ7NNHN65MA, seg_pos={0 0}, upload=441454]: Uploading piece...
Dec 26 14:32:07.855952670 (503h3m7.727035705s) [upload=441454]: Next returned nil segment
Dec 26 14:32:07.855992536 (503h3m7.727075571s) [upload=441454]: Waiting for error group managing segments...
Dec 26 14:32:08.243844016 (503h3m8.114927051s) [noise=true , address=145.79.227.3:4153 , piece_id=PHL7TPZI2LJJF3PC4YSO3UUKI7HH25LL5VSCK3CVVYSE6PN23NTQ, seg_pos={0 0}, upload=441454]: Done uploading piece. err: <nil>
Dec 26 14:32:08.271073345 (503h3m8.14215638s) [noise=true , address=145.79.227.48:4329 , piece_id=TYTZI6WQKX7U3F6WS4E5NXHZ7NDCW3EYGB3E7ZDV4YAU5642CJFA, seg_pos={0 0}, upload=441454]: Done uploading piece. err: <nil>
Dec 26 14:32:08.376723673 (503h3m8.247806708s) [noise=true , address=145.79.227.29:4950 , piece_id=J4PJRIZIVNWSLC74ZOEXKBKMZS5DWMB3VVKDR37X2A3YIAFNFM7Q, seg_pos={0 0}, upload=441454]: Done uploading piece. err: <nil>
Dec 26 14:32:08.487361387 (503h3m8.358444422s) [noise=true , address=193.149.164.143:4700 , piece_id=ZEGVK3MHJK364MPDTSQDNNKXPJRBUIZMPRNPYBYKSEDMOGPE2KTA, seg_pos={0 0}, upload=441454]: Done uploading piece. err: <nil>
Dec 26 14:32:08.490228667 (503h3m8.361311702s) [noise=true , address=45.91.116.9:4265 , piece_id=AUB7F6JX3PTYMDCU5HRXYLPQSSEQ4HLP2X7Z6CTJWRYA2VKEP7NQ, seg_pos={0 0}, upload=441454]: Done uploading piece. err: <nil>
Dec 26 14:32:08.572091940 (503h3m8.443174985s) [noise=true , address=77.90.31.12:4604 , piece_id=XBQP5JFWRJGF3R3HBTBJSWHTU6TDXFTZZRLO2DCPBD6LTBIVCXGA, seg_pos={0 0}, upload=441454]: Done uploading piece. err: <nil>
Dec 26 14:32:08.600147783 (503h3m8.471230829s) [noise=true , address=77.90.31.19:4863 , piece_id=JIERBYN26O3X36AUJLU4OUHY22GQE3YWQUOTQ5N3KNGQVAR4O27A, seg_pos={0 0}, upload=441454]: Done uploading piece. err: <nil>
Dec 26 14:32:08.982897218 (503h3m8.853980253s) [noise=true , address=77.90.31.18:4824 , piece_id=DMVU5JF44YYBFO6W57RLOVDSGMK7RCIBGKSRP5WZUWALNZFZHEMA, seg_pos={0 0}, upload=441454]: Done uploading piece. err: <nil>
Dec 26 14:32:09.220883107 (503h3m9.091966152s) [noise=true , address=77.90.31.13:4633 , piece_id=A2EPWATZFDQ4S3UV6SKFERNXSAMFNQ7VRUAUKRNHJWGDYJ7Q6FBQ, seg_pos={0 0}, upload=441454]: Done uploading piece. err: <nil>
Dec 26 14:32:09.238012875 (503h3m9.10909592s) [noise=true , address=145.79.227.44:4197 , piece_id=3T7KOR4LKB7IQELOZMTZU4STM2MCHGQ7GN4O2PJHY5HQJTH7Q4OA, seg_pos={0 0}, upload=441454]: Done uploading piece. err: <nil>
Dec 26 14:32:09.275916891 (503h3m9.146999926s) [noise=true , address=145.79.227.30:4794 , piece_id=2SAB6NZV3SRNENOPYFOAQZIR3TK7CR2XV6YBZYGYQSWW3G4TFV4Q, seg_pos={0 0}, upload=441454]: Done uploading piece. err: <nil>
Dec 26 14:32:09.443175823 (503h3m9.314258878s) [noise=true , address=77.90.31.14:4695 , piece_id=LX2V6PMVTQPMIVIRRM7FIWYC6FLW2NTCVVRZ4QDSETZ7NNHN65MA, seg_pos={0 0}, upload=441454]: Done uploading piece. err: <nil>
Dec 26 14:32:09.628270297 (503h3m9.499353342s) [noise=true , address=77.90.31.11:4580 , piece_id=EPGIESRMH2ZMNMOEPKV3WLRHZ7DMHSDYP45GTOCD4VV4NS4ZQ6PQ, seg_pos={0 0}, upload=441454]: Done uploading piece. err: <nil>
Dec 26 14:32:09.676678414 (503h3m9.547761449s) [noise=true , address=145.79.227.40:4068 , piece_id=TBIDPYRQBO7QAUEEGJYLA5YQD5YSIJYWIXHO6KUMFRM2WW3KRCLQ, seg_pos={0 0}, upload=441454]: Done uploading piece. err: <nil>
Dec 26 14:32:09.676705266 (503h3m9.547788311s) [seg_pos={0 0}, upload=441454]: Segment reached optimal threshold of 14 pieces.
Dec 26 14:32:09.676924543 (503h3m9.548007588s) [noise=true , address=77.90.31.15:4711 , piece_id=VY5TQ2WLI2XJZ2RJ2UNT2VA6E7S6U2DYF243MWK7N7AAOJWIDF4A, seg_pos={0 0}, upload=441454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu): protocol: write tcp 188.166.225.170:33416->77.90.31.15:4711: use of closed network connection; write tcp 188.166.225.170:33416->77.90.31.15:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33416->77.90.31.15:4711: use of closed network connection
Dec 26 14:32:09.676935634 (503h3m9.548018679s) [noise=true , address=77.90.31.15:4711 , piece_id=VY5TQ2WLI2XJZ2RJ2UNT2VA6E7S6U2DYF243MWK7N7AAOJWIDF4A, seg_pos={0 0}, upload=441454]: Long tail cancellation due to optimal threshold
Dec 26 14:32:09.676969679 (503h3m9.548052724s) [noise=true , address=145.79.227.34:4936 , piece_id=IJWHOOUSWAJ3MWMAHIYYJSCK5C5BX4DJM4GDQMQDZPP7TG6GGFTQ, seg_pos={0 0}, upload=441454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DicJh1hd5NUoj9uxoqzeaCmjGHwfk2VUaA9zQFmv72wBdBoW8): protocol: expected piece hash; context canceled; EOF
Dec 26 14:32:09.676978005 (503h3m9.54806107s) [noise=true , address=145.79.227.34:4936 , piece_id=IJWHOOUSWAJ3MWMAHIYYJSCK5C5BX4DJM4GDQMQDZPP7TG6GGFTQ, seg_pos={0 0}, upload=441454]: Long tail cancellation due to optimal threshold
Dec 26 14:32:09.677098184 (503h3m9.548181249s) [noise=true , address=45.91.116.4:4967 , piece_id=AJWSUN2SN4FD3EE3FNISARGF34G5QRW76KBHXQSMI2AJJ33J5FPQ, seg_pos={0 0}, upload=441454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd): protocol: expected piece hash; context canceled; EOF
Dec 26 14:32:09.677107842 (503h3m9.548190887s) [noise=true , address=45.91.116.4:4967 , piece_id=AJWSUN2SN4FD3EE3FNISARGF34G5QRW76KBHXQSMI2AJJ33J5FPQ, seg_pos={0 0}, upload=441454]: Long tail cancellation due to optimal threshold
Dec 26 14:32:09.677318984 (503h3m9.548402019s) [noise=true , address=64.49.13.169:4716 , piece_id=B3MX3TD6WCWHUSEJAHIUQWBQJJMY7M4ELXZWC6DGK5UG734HZU4A, seg_pos={0 0}, upload=441454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:47730->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:47730->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47730->64.49.13.169:4716: use of closed network connection
Dec 26 14:32:09.677328773 (503h3m9.548411808s) [noise=true , address=64.49.13.169:4716 , piece_id=B3MX3TD6WCWHUSEJAHIUQWBQJJMY7M4ELXZWC6DGK5UG734HZU4A, seg_pos={0 0}, upload=441454]: Long tail cancellation due to optimal threshold
Dec 26 14:32:09.677350744 (503h3m9.548433789s) [seg_pos={0 0}, upload=441454]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 14:32:09.677433893 (503h3m9.548516938s) [upload=441454]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:32:09.677440095 (503h3m9.54852314s) [upload=441454]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:32:09.969560615 (503h3m9.84064365s) [upload=441454]: Done uploading segments...
Dec 26 14:32:09.969584000 (503h3m9.840667045s) [upload=441454]: Upload finished. err: <nil>
Duration: 48.395882509s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
54 out of 60 is failed (threshold=6) [.XXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX.XXXXX.XXXXXXX.XXXXXXXXX.XX] duration:
5.001916923s trace:
https://jaeger.dev.storj.io/trace/42014ba62a8f73f7
FAIL (2025-12-26 14:33:43 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001943813s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
13.378426752s trace:
https://jaeger.dev.storj.io/trace/3ccdb98964ad100c
FAIL (2025-12-26 14:33:42 UTC)
orbiter warning: context deadline exceeded
Duration: 13.378453784s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
13.05249232s trace:
https://jaeger.dev.storj.io/trace/5b63846e68ea1b18
FAIL (2025-12-26 14:33:39 UTC)
orbiter warning: context deadline exceeded
Duration: 13.052516145s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
13.566602831s trace:
https://jaeger.dev.storj.io/trace/3ccdb98964ad100c
FAIL (2025-12-26 14:33:15 UTC)
orbiter warning: context deadline exceeded
Duration: 13.566635162s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
54 out of 60 is failed (threshold=6) [X.XXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX.XXXXX.XXXXXXX.XXXXXXXXX.X] duration:
5.000691612s trace:
https://jaeger.dev.storj.io/trace/1a5f773279d21d45
FAIL (2025-12-26 14:33:07 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.00071694s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [..XX...X...X...........X..X.X...........X....XX.XXX..XXX.X..] duration:
5.823436233s trace:
https://jaeger.dev.storj.io/trace/4957367a1f492fcb
FAIL (2025-12-26 14:33:05 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 14:31:33.597437556 (503h2m33.468520601s) [upload=441449]: Starting upload
Dec 26 14:31:33.597553948 (503h2m33.468636993s) [upload=441449]: Done starting upload
Dec 26 14:31:33.597603222 (503h2m33.468686267s) [upload=441449]: Uploading segments...
Dec 26 14:31:33.597789576 (503h2m33.468872611s) [upload=441449]: Got next segment. Inline: false
Dec 26 14:31:33.597882563 (503h2m33.468965598s) [upload=441449]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 14:31:33.597889186 (503h2m33.468972231s) [upload=441449]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 14:31:33.846933082 (503h2m33.718016127s) [seg_pos={0 0}, upload=441449]: Begin upload segment...
Dec 26 14:31:33.846983659 (503h2m33.718066694s) [seg_pos={0 0}, upload=441449]: Done begin upload segment.
Dec 26 14:31:33.847017223 (503h2m33.718100268s) [noise=true , address=145.79.227.51:4409 , piece_id=DL3T4T3DOTHIPXAZ5BEEM42OCD3FKTU5XOHVWNDIUQE3ERZFLWZQ, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.847050697 (503h2m33.718133732s) [noise=true , address=77.90.31.14:4661 , piece_id=YFYXETEUCFVK7MBM3QCVVSE4FR4SHJBHRZLD4WHTI64RDSGMGQNQ, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.847570256 (503h2m33.718653311s) [noise=true , address=77.90.31.17:4773 , piece_id=FHH435DDQIZKXMHG76N47EK7JD5XC3IOXFUSSFPDFMKBOLBN2OKA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.848112698 (503h2m33.719195744s) [noise=true , address=45.91.116.4:4940 , piece_id=5X6DK3LI6ASO6EYUS6UTWNZDOR7U3IMVBBY6M73IEAZ52RWS2HYA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.848499495 (503h2m33.71958254s) [noise=true , address=145.79.227.49:4337 , piece_id=JVMK4NXGKUWEP2XAAZZWTSE5TEPFYVGKE3FFWDMOVHRK6Z27K4RA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.848875521 (503h2m33.719958556s) [noise=true , address=145.79.227.45:4229 , piece_id=5RKLGUD2ZO5JEAEK3GVEKTKBHYDVOQ5BK5NYJZQUB7P2I5NBFDBA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.849364823 (503h2m33.720447868s) [noise=true , address=145.79.227.21:4473 , piece_id=UCQK72VUEHSCCJBIEZJ4UD7SMFZUPXK4KM6VGB47RPMOOD2HVTHQ, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.849763733 (503h2m33.720846768s) [noise=true , address=45.91.116.14:4190 , piece_id=ZGP23KORYXHV2ZNPQSDTLCR7PR4HCYPVJ7QCTA6DBYHRSNJAGXPA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.850922750 (503h2m33.722005795s) [noise=true , address=145.79.227.36:4518 , piece_id=LJXTQWBSIVHP5JDCT6PEWOICJFW72MGSRPBOYDIYCJEKRRMLGOLA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.851242398 (503h2m33.722325443s) [noise=true , address=45.91.116.5:4298 , piece_id=6AY2UYXYETFVFKWHNY7OHUONOYAMA26N2YF3RHAYFA2ZZW2ZNQ3Q, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.852200212 (503h2m33.723283247s) [noise=true , address=145.79.227.47:4266 , piece_id=VRNGVBKFWQRJA7TAR6QZ52JVFLKTCNBOYK7VSZSUUJVPFAJBA6GA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.853435684 (503h2m33.72451873s) [noise=true , address=145.79.227.38:4979 , piece_id=52FH5WDZ76TN3LRN6IMMTS5OE3TXC637YY53IAEU7MHGHRE7R7JA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.854620921 (503h2m33.725703956s) [noise=true , address=145.79.227.50:4375 , piece_id=23RC4QNCFLTCIF4JHVD7ESHDIDW2CWWKCSE5Z64YX3P5526GDCJA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.856049742 (503h2m33.727132787s) [noise=true , address=77.90.31.19:4844 , piece_id=O56QQ4AUPE24ERTZQYUQEOK6NYNWYI3DNZQ5FWKFEZLLHF63OANQ, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.857977794 (503h2m33.729060839s) [noise=true , address=77.90.31.11:4555 , piece_id=W7NU2VVWGDDND474EICGZYS3GPOR75FNBD7Q6ZVIZVEQ2CRYCNKQ, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.858982477 (503h2m33.730065512s) [noise=true , address=145.79.227.29:4943 , piece_id=Y7JPU43H23ZITBIQ6TK7RUZPG3T3NMGPDTB7S6DJNZG7IVGTUYVA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:35.571477638 (503h2m35.442560673s) [upload=441449]: Next returned nil segment
Dec 26 14:31:35.571490583 (503h2m35.442573628s) [upload=441449]: Waiting for error group managing segments...
Dec 26 14:31:35.855026257 (503h2m35.726109302s) [noise=true , address=145.79.227.49:4337 , piece_id=JVMK4NXGKUWEP2XAAZZWTSE5TEPFYVGKE3FFWDMOVHRK6Z27K4RA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:35.862095689 (503h2m35.733178734s) [noise=true , address=145.79.227.38:4979 , piece_id=52FH5WDZ76TN3LRN6IMMTS5OE3TXC637YY53IAEU7MHGHRE7R7JA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:35.862796403 (503h2m35.733879438s) [noise=true , address=145.79.227.36:4518 , piece_id=LJXTQWBSIVHP5JDCT6PEWOICJFW72MGSRPBOYDIYCJEKRRMLGOLA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:35.864289396 (503h2m35.735372431s) [noise=true , address=145.79.227.50:4375 , piece_id=23RC4QNCFLTCIF4JHVD7ESHDIDW2CWWKCSE5Z64YX3P5526GDCJA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:35.869609206 (503h2m35.740692251s) [noise=true , address=145.79.227.45:4229 , piece_id=5RKLGUD2ZO5JEAEK3GVEKTKBHYDVOQ5BK5NYJZQUB7P2I5NBFDBA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:35.872899100 (503h2m35.743982135s) [noise=true , address=145.79.227.21:4473 , piece_id=UCQK72VUEHSCCJBIEZJ4UD7SMFZUPXK4KM6VGB47RPMOOD2HVTHQ, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:35.881408674 (503h2m35.752491709s) [noise=true , address=145.79.227.29:4943 , piece_id=Y7JPU43H23ZITBIQ6TK7RUZPG3T3NMGPDTB7S6DJNZG7IVGTUYVA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:35.888713384 (503h2m35.759796429s) [noise=true , address=145.79.227.51:4409 , piece_id=DL3T4T3DOTHIPXAZ5BEEM42OCD3FKTU5XOHVWNDIUQE3ERZFLWZQ, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:35.893099275 (503h2m35.76418231s) [noise=true , address=145.79.227.47:4266 , piece_id=VRNGVBKFWQRJA7TAR6QZ52JVFLKTCNBOYK7VSZSUUJVPFAJBA6GA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:36.043138096 (503h2m35.914221141s) [noise=true , address=77.90.31.14:4661 , piece_id=YFYXETEUCFVK7MBM3QCVVSE4FR4SHJBHRZLD4WHTI64RDSGMGQNQ, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:36.087256205 (503h2m35.95833924s) [noise=true , address=45.91.116.5:4298 , piece_id=6AY2UYXYETFVFKWHNY7OHUONOYAMA26N2YF3RHAYFA2ZZW2ZNQ3Q, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:36.233410791 (503h2m36.104493836s) [noise=true , address=45.91.116.14:4190 , piece_id=ZGP23KORYXHV2ZNPQSDTLCR7PR4HCYPVJ7QCTA6DBYHRSNJAGXPA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:36.993343372 (503h2m36.864426407s) [noise=true , address=77.90.31.19:4844 , piece_id=O56QQ4AUPE24ERTZQYUQEOK6NYNWYI3DNZQ5FWKFEZLLHF63OANQ, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:36.993368200 (503h2m36.864451285s) [seg_pos={0 0}, upload=441449]: Segment reached optimal threshold of 13 pieces.
Dec 26 14:31:36.993595764 (503h2m36.864678799s) [noise=true , address=77.90.31.11:4555 , piece_id=W7NU2VVWGDDND474EICGZYS3GPOR75FNBD7Q6ZVIZVEQ2CRYCNKQ, seg_pos={0 0}, upload=441449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: write tcp 188.166.225.170:46760->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:46760->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46760->77.90.31.11:4555: use of closed network connection
Dec 26 14:31:36.993605442 (503h2m36.864688477s) [noise=true , address=77.90.31.11:4555 , piece_id=W7NU2VVWGDDND474EICGZYS3GPOR75FNBD7Q6ZVIZVEQ2CRYCNKQ, seg_pos={0 0}, upload=441449]: Long tail cancellation due to optimal threshold
Dec 26 14:31:36.993751880 (503h2m36.864834946s) [noise=true , address=77.90.31.17:4773 , piece_id=FHH435DDQIZKXMHG76N47EK7JD5XC3IOXFUSSFPDFMKBOLBN2OKA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF
Dec 26 14:31:36.993770246 (503h2m36.864853291s) [noise=true , address=77.90.31.17:4773 , piece_id=FHH435DDQIZKXMHG76N47EK7JD5XC3IOXFUSSFPDFMKBOLBN2OKA, seg_pos={0 0}, upload=441449]: Long tail cancellation due to optimal threshold
Dec 26 14:31:36.993934509 (503h2m36.865017544s) [noise=true , address=45.91.116.4:4940 , piece_id=5X6DK3LI6ASO6EYUS6UTWNZDOR7U3IMVBBY6M73IEAZ52RWS2HYA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF
Dec 26 14:31:36.993942774 (503h2m36.865025809s) [noise=true , address=45.91.116.4:4940 , piece_id=5X6DK3LI6ASO6EYUS6UTWNZDOR7U3IMVBBY6M73IEAZ52RWS2HYA, seg_pos={0 0}, upload=441449]: Long tail cancellation due to optimal threshold
Dec 26 14:31:36.993961339 (503h2m36.865044384s) [seg_pos={0 0}, upload=441449]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 14:31:36.994038988 (503h2m36.865122033s) [upload=441449]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:31:36.994045901 (503h2m36.865128966s) [upload=441449]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:31:37.298152335 (503h2m37.16923537s) [upload=441449]: Done uploading segments...
Dec 26 14:31:37.298171903 (503h2m37.169254938s) [upload=441449]: Upload finished. err: <nil>
Duration: 5.823468964s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
19 out of 60 is failed (threshold=3) [.XX.....X..........X..X..X.X.X....X.....XXX.XX.XX.....X...XX] duration:
1m0.002590355s trace:
https://jaeger.dev.storj.io/trace/6da2926055e33842
FAIL (2025-12-26 14:33:05 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12qXhyMbfXELW1Vg5z9NpqFX1fm3v2pGEaA56qSc8kHefPNNNYs, Piece ID: A2EPWATZFDQ4S3UV6SKFERNXSAMFNQ7VRUAUKRNHJWGDYJ7Q6FBQ) context deadline exceeded; piecestore: (Node ID: 12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D, Piece ID: EPGIESRMH2ZMNMOEPKV3WLRHZ7DMHSDYP45GTOCD4VV4NS4ZQ6PQ) context deadline exceeded; piecestore: (Node ID: 1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY, Piece ID: ZEGVK3MHJK364MPDTSQDNNKXPJRBUIZMPRNPYBYKSEDMOGPE2KTA) context deadline exceeded; piecestore: (Node ID: 1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, Piece ID: XBQP5JFWRJGF3R3HBTBJSWHTU6TDXFTZZRLO2DCPBD6LTBIVCXGA) context deadline exceeded; piecestore: (Node ID: 124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, Piece ID: JIERBYN26O3X36AUJLU4OUHY22GQE3YWQUOTQ5N3KNGQVAR4O27A) context deadline exceeded
======LOGS======
piece...
Dec 26 14:32:06.384298372 (503h3m6.255381407s) [noise=true , address=77.90.31.19:4863 , piece_id=JIERBYN26O3X36AUJLU4OUHY22GQE3YWQUOTQ5N3KNGQVAR4O27A, seg_pos={0 0}, upload=441454]: Uploading piece...
Dec 26 14:32:06.384615456 (503h3m6.255698501s) [noise=true , address=77.90.31.11:4580 , piece_id=EPGIESRMH2ZMNMOEPKV3WLRHZ7DMHSDYP45GTOCD4VV4NS4ZQ6PQ, seg_pos={0 0}, upload=441454]: Uploading piece...
Dec 26 14:32:06.384930195 (503h3m6.25601324s) [noise=true , address=77.90.31.15:4711 , piece_id=VY5TQ2WLI2XJZ2RJ2UNT2VA6E7S6U2DYF243MWK7N7AAOJWIDF4A, seg_pos={0 0}, upload=441454]: Uploading piece...
Dec 26 14:32:06.385356466 (503h3m6.256439511s) [noise=true , address=45.91.116.4:4967 , piece_id=AJWSUN2SN4FD3EE3FNISARGF34G5QRW76KBHXQSMI2AJJ33J5FPQ, seg_pos={0 0}, upload=441454]: Uploading piece...
Dec 26 14:32:06.385694320 (503h3m6.256777365s) [noise=true , address=77.90.31.12:4604 , piece_id=XBQP5JFWRJGF3R3HBTBJSWHTU6TDXFTZZRLO2DCPBD6LTBIVCXGA, seg_pos={0 0}, upload=441454]: Uploading piece...
Dec 26 14:32:06.386092868 (503h3m6.257175903s) [noise=true , address=145.79.227.3:4153 , piece_id=PHL7TPZI2LJJF3PC4YSO3UUKI7HH25LL5VSCK3CVVYSE6PN23NTQ, seg_pos={0 0}, upload=441454]: Uploading piece...
Dec 26 14:32:06.387802996 (503h3m6.258886041s) [noise=true , address=145.79.227.40:4068 , piece_id=TBIDPYRQBO7QAUEEGJYLA5YQD5YSIJYWIXHO6KUMFRM2WW3KRCLQ, seg_pos={0 0}, upload=441454]: Uploading piece...
Dec 26 14:32:06.389673156 (503h3m6.260756201s) [noise=true , address=193.149.164.143:4700 , piece_id=ZEGVK3MHJK364MPDTSQDNNKXPJRBUIZMPRNPYBYKSEDMOGPE2KTA, seg_pos={0 0}, upload=441454]: Uploading piece...
Dec 26 14:32:06.391484555 (503h3m6.2625676s) [noise=true , address=64.49.13.169:4716 , piece_id=B3MX3TD6WCWHUSEJAHIUQWBQJJMY7M4ELXZWC6DGK5UG734HZU4A, seg_pos={0 0}, upload=441454]: Uploading piece...
Dec 26 14:32:06.393370396 (503h3m6.264453441s) [noise=true , address=77.90.31.18:4824 , piece_id=DMVU5JF44YYBFO6W57RLOVDSGMK7RCIBGKSRP5WZUWALNZFZHEMA, seg_pos={0 0}, upload=441454]: Uploading piece...
Dec 26 14:32:06.395059202 (503h3m6.266142247s) [noise=true , address=145.79.227.44:4197 , piece_id=3T7KOR4LKB7IQELOZMTZU4STM2MCHGQ7GN4O2PJHY5HQJTH7Q4OA, seg_pos={0 0}, upload=441454]: Uploading piece...
Dec 26 14:32:06.397013243 (503h3m6.268096288s) [noise=true , address=145.79.227.30:4794 , piece_id=2SAB6NZV3SRNENOPYFOAQZIR3TK7CR2XV6YBZYGYQSWW3G4TFV4Q, seg_pos={0 0}, upload=441454]: Uploading piece...
Dec 26 14:32:06.399007981 (503h3m6.270091026s) [noise=true , address=145.79.227.34:4936 , piece_id=IJWHOOUSWAJ3MWMAHIYYJSCK5C5BX4DJM4GDQMQDZPP7TG6GGFTQ, seg_pos={0 0}, upload=441454]: Uploading piece...
Dec 26 14:32:06.400585856 (503h3m6.271668901s) [noise=true , address=145.79.227.48:4329 , piece_id=TYTZI6WQKX7U3F6WS4E5NXHZ7NDCW3EYGB3E7ZDV4YAU5642CJFA, seg_pos={0 0}, upload=441454]: Uploading piece...
Dec 26 14:32:06.402088056 (503h3m6.273171102s) [noise=true , address=77.90.31.14:4695 , piece_id=LX2V6PMVTQPMIVIRRM7FIWYC6FLW2NTCVVRZ4QDSETZ7NNHN65MA, seg_pos={0 0}, upload=441454]: Uploading piece...
Dec 26 14:32:07.855952670 (503h3m7.727035705s) [upload=441454]: Next returned nil segment
Dec 26 14:32:07.855992536 (503h3m7.727075571s) [upload=441454]: Waiting for error group managing segments...
Dec 26 14:32:08.243844016 (503h3m8.114927051s) [noise=true , address=145.79.227.3:4153 , piece_id=PHL7TPZI2LJJF3PC4YSO3UUKI7HH25LL5VSCK3CVVYSE6PN23NTQ, seg_pos={0 0}, upload=441454]: Done uploading piece. err: <nil>
Dec 26 14:32:08.271073345 (503h3m8.14215638s) [noise=true , address=145.79.227.48:4329 , piece_id=TYTZI6WQKX7U3F6WS4E5NXHZ7NDCW3EYGB3E7ZDV4YAU5642CJFA, seg_pos={0 0}, upload=441454]: Done uploading piece. err: <nil>
Dec 26 14:32:08.376723673 (503h3m8.247806708s) [noise=true , address=145.79.227.29:4950 , piece_id=J4PJRIZIVNWSLC74ZOEXKBKMZS5DWMB3VVKDR37X2A3YIAFNFM7Q, seg_pos={0 0}, upload=441454]: Done uploading piece. err: <nil>
Dec 26 14:32:08.487361387 (503h3m8.358444422s) [noise=true , address=193.149.164.143:4700 , piece_id=ZEGVK3MHJK364MPDTSQDNNKXPJRBUIZMPRNPYBYKSEDMOGPE2KTA, seg_pos={0 0}, upload=441454]: Done uploading piece. err: <nil>
Dec 26 14:32:08.490228667 (503h3m8.361311702s) [noise=true , address=45.91.116.9:4265 , piece_id=AUB7F6JX3PTYMDCU5HRXYLPQSSEQ4HLP2X7Z6CTJWRYA2VKEP7NQ, seg_pos={0 0}, upload=441454]: Done uploading piece. err: <nil>
Dec 26 14:32:08.572091940 (503h3m8.443174985s) [noise=true , address=77.90.31.12:4604 , piece_id=XBQP5JFWRJGF3R3HBTBJSWHTU6TDXFTZZRLO2DCPBD6LTBIVCXGA, seg_pos={0 0}, upload=441454]: Done uploading piece. err: <nil>
Dec 26 14:32:08.600147783 (503h3m8.471230829s) [noise=true , address=77.90.31.19:4863 , piece_id=JIERBYN26O3X36AUJLU4OUHY22GQE3YWQUOTQ5N3KNGQVAR4O27A, seg_pos={0 0}, upload=441454]: Done uploading piece. err: <nil>
Dec 26 14:32:08.982897218 (503h3m8.853980253s) [noise=true , address=77.90.31.18:4824 , piece_id=DMVU5JF44YYBFO6W57RLOVDSGMK7RCIBGKSRP5WZUWALNZFZHEMA, seg_pos={0 0}, upload=441454]: Done uploading piece. err: <nil>
Dec 26 14:32:09.220883107 (503h3m9.091966152s) [noise=true , address=77.90.31.13:4633 , piece_id=A2EPWATZFDQ4S3UV6SKFERNXSAMFNQ7VRUAUKRNHJWGDYJ7Q6FBQ, seg_pos={0 0}, upload=441454]: Done uploading piece. err: <nil>
Dec 26 14:32:09.238012875 (503h3m9.10909592s) [noise=true , address=145.79.227.44:4197 , piece_id=3T7KOR4LKB7IQELOZMTZU4STM2MCHGQ7GN4O2PJHY5HQJTH7Q4OA, seg_pos={0 0}, upload=441454]: Done uploading piece. err: <nil>
Dec 26 14:32:09.275916891 (503h3m9.146999926s) [noise=true , address=145.79.227.30:4794 , piece_id=2SAB6NZV3SRNENOPYFOAQZIR3TK7CR2XV6YBZYGYQSWW3G4TFV4Q, seg_pos={0 0}, upload=441454]: Done uploading piece. err: <nil>
Dec 26 14:32:09.443175823 (503h3m9.314258878s) [noise=true , address=77.90.31.14:4695 , piece_id=LX2V6PMVTQPMIVIRRM7FIWYC6FLW2NTCVVRZ4QDSETZ7NNHN65MA, seg_pos={0 0}, upload=441454]: Done uploading piece. err: <nil>
Dec 26 14:32:09.628270297 (503h3m9.499353342s) [noise=true , address=77.90.31.11:4580 , piece_id=EPGIESRMH2ZMNMOEPKV3WLRHZ7DMHSDYP45GTOCD4VV4NS4ZQ6PQ, seg_pos={0 0}, upload=441454]: Done uploading piece. err: <nil>
Dec 26 14:32:09.676678414 (503h3m9.547761449s) [noise=true , address=145.79.227.40:4068 , piece_id=TBIDPYRQBO7QAUEEGJYLA5YQD5YSIJYWIXHO6KUMFRM2WW3KRCLQ, seg_pos={0 0}, upload=441454]: Done uploading piece. err: <nil>
Dec 26 14:32:09.676705266 (503h3m9.547788311s) [seg_pos={0 0}, upload=441454]: Segment reached optimal threshold of 14 pieces.
Dec 26 14:32:09.676924543 (503h3m9.548007588s) [noise=true , address=77.90.31.15:4711 , piece_id=VY5TQ2WLI2XJZ2RJ2UNT2VA6E7S6U2DYF243MWK7N7AAOJWIDF4A, seg_pos={0 0}, upload=441454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu): protocol: write tcp 188.166.225.170:33416->77.90.31.15:4711: use of closed network connection; write tcp 188.166.225.170:33416->77.90.31.15:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33416->77.90.31.15:4711: use of closed network connection
Dec 26 14:32:09.676935634 (503h3m9.548018679s) [noise=true , address=77.90.31.15:4711 , piece_id=VY5TQ2WLI2XJZ2RJ2UNT2VA6E7S6U2DYF243MWK7N7AAOJWIDF4A, seg_pos={0 0}, upload=441454]: Long tail cancellation due to optimal threshold
Dec 26 14:32:09.676969679 (503h3m9.548052724s) [noise=true , address=145.79.227.34:4936 , piece_id=IJWHOOUSWAJ3MWMAHIYYJSCK5C5BX4DJM4GDQMQDZPP7TG6GGFTQ, seg_pos={0 0}, upload=441454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DicJh1hd5NUoj9uxoqzeaCmjGHwfk2VUaA9zQFmv72wBdBoW8): protocol: expected piece hash; context canceled; EOF
Dec 26 14:32:09.676978005 (503h3m9.54806107s) [noise=true , address=145.79.227.34:4936 , piece_id=IJWHOOUSWAJ3MWMAHIYYJSCK5C5BX4DJM4GDQMQDZPP7TG6GGFTQ, seg_pos={0 0}, upload=441454]: Long tail cancellation due to optimal threshold
Dec 26 14:32:09.677098184 (503h3m9.548181249s) [noise=true , address=45.91.116.4:4967 , piece_id=AJWSUN2SN4FD3EE3FNISARGF34G5QRW76KBHXQSMI2AJJ33J5FPQ, seg_pos={0 0}, upload=441454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd): protocol: expected piece hash; context canceled; EOF
Dec 26 14:32:09.677107842 (503h3m9.548190887s) [noise=true , address=45.91.116.4:4967 , piece_id=AJWSUN2SN4FD3EE3FNISARGF34G5QRW76KBHXQSMI2AJJ33J5FPQ, seg_pos={0 0}, upload=441454]: Long tail cancellation due to optimal threshold
Dec 26 14:32:09.677318984 (503h3m9.548402019s) [noise=true , address=64.49.13.169:4716 , piece_id=B3MX3TD6WCWHUSEJAHIUQWBQJJMY7M4ELXZWC6DGK5UG734HZU4A, seg_pos={0 0}, upload=441454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:47730->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:47730->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47730->64.49.13.169:4716: use of closed network connection
Dec 26 14:32:09.677328773 (503h3m9.548411808s) [noise=true , address=64.49.13.169:4716 , piece_id=B3MX3TD6WCWHUSEJAHIUQWBQJJMY7M4ELXZWC6DGK5UG734HZU4A, seg_pos={0 0}, upload=441454]: Long tail cancellation due to optimal threshold
Dec 26 14:32:09.677350744 (503h3m9.548433789s) [seg_pos={0 0}, upload=441454]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 14:32:09.677433893 (503h3m9.548516938s) [upload=441454]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:32:09.677440095 (503h3m9.54852314s) [upload=441454]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:32:09.969560615 (503h3m9.84064365s) [upload=441454]: Done uploading segments...
Dec 26 14:32:09.969584000 (503h3m9.840667045s) [upload=441454]: Upload finished. err: <nil>
Duration: 1m0.002602007s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
20.2395529s trace:
https://jaeger.dev.storj.io/trace/5b63846e68ea1b18
FAIL (2025-12-26 14:32:50 UTC)
orbiter warning: context deadline exceeded
Duration: 20.239572227s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [X..XX...X...X...........X..X.X...........X....XX.XXX..XXX.X.] duration:
5.771132842s trace:
https://jaeger.dev.storj.io/trace/4957367a1f492fcb
FAIL (2025-12-26 14:32:37 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 14:31:33.597437556 (503h2m33.468520601s) [upload=441449]: Starting upload
Dec 26 14:31:33.597553948 (503h2m33.468636993s) [upload=441449]: Done starting upload
Dec 26 14:31:33.597603222 (503h2m33.468686267s) [upload=441449]: Uploading segments...
Dec 26 14:31:33.597789576 (503h2m33.468872611s) [upload=441449]: Got next segment. Inline: false
Dec 26 14:31:33.597882563 (503h2m33.468965598s) [upload=441449]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 14:31:33.597889186 (503h2m33.468972231s) [upload=441449]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 14:31:33.846933082 (503h2m33.718016127s) [seg_pos={0 0}, upload=441449]: Begin upload segment...
Dec 26 14:31:33.846983659 (503h2m33.718066694s) [seg_pos={0 0}, upload=441449]: Done begin upload segment.
Dec 26 14:31:33.847017223 (503h2m33.718100268s) [noise=true , address=145.79.227.51:4409 , piece_id=DL3T4T3DOTHIPXAZ5BEEM42OCD3FKTU5XOHVWNDIUQE3ERZFLWZQ, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.847050697 (503h2m33.718133732s) [noise=true , address=77.90.31.14:4661 , piece_id=YFYXETEUCFVK7MBM3QCVVSE4FR4SHJBHRZLD4WHTI64RDSGMGQNQ, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.847570256 (503h2m33.718653311s) [noise=true , address=77.90.31.17:4773 , piece_id=FHH435DDQIZKXMHG76N47EK7JD5XC3IOXFUSSFPDFMKBOLBN2OKA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.848112698 (503h2m33.719195744s) [noise=true , address=45.91.116.4:4940 , piece_id=5X6DK3LI6ASO6EYUS6UTWNZDOR7U3IMVBBY6M73IEAZ52RWS2HYA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.848499495 (503h2m33.71958254s) [noise=true , address=145.79.227.49:4337 , piece_id=JVMK4NXGKUWEP2XAAZZWTSE5TEPFYVGKE3FFWDMOVHRK6Z27K4RA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.848875521 (503h2m33.719958556s) [noise=true , address=145.79.227.45:4229 , piece_id=5RKLGUD2ZO5JEAEK3GVEKTKBHYDVOQ5BK5NYJZQUB7P2I5NBFDBA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.849364823 (503h2m33.720447868s) [noise=true , address=145.79.227.21:4473 , piece_id=UCQK72VUEHSCCJBIEZJ4UD7SMFZUPXK4KM6VGB47RPMOOD2HVTHQ, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.849763733 (503h2m33.720846768s) [noise=true , address=45.91.116.14:4190 , piece_id=ZGP23KORYXHV2ZNPQSDTLCR7PR4HCYPVJ7QCTA6DBYHRSNJAGXPA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.850922750 (503h2m33.722005795s) [noise=true , address=145.79.227.36:4518 , piece_id=LJXTQWBSIVHP5JDCT6PEWOICJFW72MGSRPBOYDIYCJEKRRMLGOLA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.851242398 (503h2m33.722325443s) [noise=true , address=45.91.116.5:4298 , piece_id=6AY2UYXYETFVFKWHNY7OHUONOYAMA26N2YF3RHAYFA2ZZW2ZNQ3Q, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.852200212 (503h2m33.723283247s) [noise=true , address=145.79.227.47:4266 , piece_id=VRNGVBKFWQRJA7TAR6QZ52JVFLKTCNBOYK7VSZSUUJVPFAJBA6GA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.853435684 (503h2m33.72451873s) [noise=true , address=145.79.227.38:4979 , piece_id=52FH5WDZ76TN3LRN6IMMTS5OE3TXC637YY53IAEU7MHGHRE7R7JA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.854620921 (503h2m33.725703956s) [noise=true , address=145.79.227.50:4375 , piece_id=23RC4QNCFLTCIF4JHVD7ESHDIDW2CWWKCSE5Z64YX3P5526GDCJA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.856049742 (503h2m33.727132787s) [noise=true , address=77.90.31.19:4844 , piece_id=O56QQ4AUPE24ERTZQYUQEOK6NYNWYI3DNZQ5FWKFEZLLHF63OANQ, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.857977794 (503h2m33.729060839s) [noise=true , address=77.90.31.11:4555 , piece_id=W7NU2VVWGDDND474EICGZYS3GPOR75FNBD7Q6ZVIZVEQ2CRYCNKQ, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.858982477 (503h2m33.730065512s) [noise=true , address=145.79.227.29:4943 , piece_id=Y7JPU43H23ZITBIQ6TK7RUZPG3T3NMGPDTB7S6DJNZG7IVGTUYVA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:35.571477638 (503h2m35.442560673s) [upload=441449]: Next returned nil segment
Dec 26 14:31:35.571490583 (503h2m35.442573628s) [upload=441449]: Waiting for error group managing segments...
Dec 26 14:31:35.855026257 (503h2m35.726109302s) [noise=true , address=145.79.227.49:4337 , piece_id=JVMK4NXGKUWEP2XAAZZWTSE5TEPFYVGKE3FFWDMOVHRK6Z27K4RA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:35.862095689 (503h2m35.733178734s) [noise=true , address=145.79.227.38:4979 , piece_id=52FH5WDZ76TN3LRN6IMMTS5OE3TXC637YY53IAEU7MHGHRE7R7JA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:35.862796403 (503h2m35.733879438s) [noise=true , address=145.79.227.36:4518 , piece_id=LJXTQWBSIVHP5JDCT6PEWOICJFW72MGSRPBOYDIYCJEKRRMLGOLA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:35.864289396 (503h2m35.735372431s) [noise=true , address=145.79.227.50:4375 , piece_id=23RC4QNCFLTCIF4JHVD7ESHDIDW2CWWKCSE5Z64YX3P5526GDCJA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:35.869609206 (503h2m35.740692251s) [noise=true , address=145.79.227.45:4229 , piece_id=5RKLGUD2ZO5JEAEK3GVEKTKBHYDVOQ5BK5NYJZQUB7P2I5NBFDBA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:35.872899100 (503h2m35.743982135s) [noise=true , address=145.79.227.21:4473 , piece_id=UCQK72VUEHSCCJBIEZJ4UD7SMFZUPXK4KM6VGB47RPMOOD2HVTHQ, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:35.881408674 (503h2m35.752491709s) [noise=true , address=145.79.227.29:4943 , piece_id=Y7JPU43H23ZITBIQ6TK7RUZPG3T3NMGPDTB7S6DJNZG7IVGTUYVA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:35.888713384 (503h2m35.759796429s) [noise=true , address=145.79.227.51:4409 , piece_id=DL3T4T3DOTHIPXAZ5BEEM42OCD3FKTU5XOHVWNDIUQE3ERZFLWZQ, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:35.893099275 (503h2m35.76418231s) [noise=true , address=145.79.227.47:4266 , piece_id=VRNGVBKFWQRJA7TAR6QZ52JVFLKTCNBOYK7VSZSUUJVPFAJBA6GA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:36.043138096 (503h2m35.914221141s) [noise=true , address=77.90.31.14:4661 , piece_id=YFYXETEUCFVK7MBM3QCVVSE4FR4SHJBHRZLD4WHTI64RDSGMGQNQ, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:36.087256205 (503h2m35.95833924s) [noise=true , address=45.91.116.5:4298 , piece_id=6AY2UYXYETFVFKWHNY7OHUONOYAMA26N2YF3RHAYFA2ZZW2ZNQ3Q, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:36.233410791 (503h2m36.104493836s) [noise=true , address=45.91.116.14:4190 , piece_id=ZGP23KORYXHV2ZNPQSDTLCR7PR4HCYPVJ7QCTA6DBYHRSNJAGXPA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:36.993343372 (503h2m36.864426407s) [noise=true , address=77.90.31.19:4844 , piece_id=O56QQ4AUPE24ERTZQYUQEOK6NYNWYI3DNZQ5FWKFEZLLHF63OANQ, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:36.993368200 (503h2m36.864451285s) [seg_pos={0 0}, upload=441449]: Segment reached optimal threshold of 13 pieces.
Dec 26 14:31:36.993595764 (503h2m36.864678799s) [noise=true , address=77.90.31.11:4555 , piece_id=W7NU2VVWGDDND474EICGZYS3GPOR75FNBD7Q6ZVIZVEQ2CRYCNKQ, seg_pos={0 0}, upload=441449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: write tcp 188.166.225.170:46760->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:46760->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46760->77.90.31.11:4555: use of closed network connection
Dec 26 14:31:36.993605442 (503h2m36.864688477s) [noise=true , address=77.90.31.11:4555 , piece_id=W7NU2VVWGDDND474EICGZYS3GPOR75FNBD7Q6ZVIZVEQ2CRYCNKQ, seg_pos={0 0}, upload=441449]: Long tail cancellation due to optimal threshold
Dec 26 14:31:36.993751880 (503h2m36.864834946s) [noise=true , address=77.90.31.17:4773 , piece_id=FHH435DDQIZKXMHG76N47EK7JD5XC3IOXFUSSFPDFMKBOLBN2OKA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF
Dec 26 14:31:36.993770246 (503h2m36.864853291s) [noise=true , address=77.90.31.17:4773 , piece_id=FHH435DDQIZKXMHG76N47EK7JD5XC3IOXFUSSFPDFMKBOLBN2OKA, seg_pos={0 0}, upload=441449]: Long tail cancellation due to optimal threshold
Dec 26 14:31:36.993934509 (503h2m36.865017544s) [noise=true , address=45.91.116.4:4940 , piece_id=5X6DK3LI6ASO6EYUS6UTWNZDOR7U3IMVBBY6M73IEAZ52RWS2HYA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF
Dec 26 14:31:36.993942774 (503h2m36.865025809s) [noise=true , address=45.91.116.4:4940 , piece_id=5X6DK3LI6ASO6EYUS6UTWNZDOR7U3IMVBBY6M73IEAZ52RWS2HYA, seg_pos={0 0}, upload=441449]: Long tail cancellation due to optimal threshold
Dec 26 14:31:36.993961339 (503h2m36.865044384s) [seg_pos={0 0}, upload=441449]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 14:31:36.994038988 (503h2m36.865122033s) [upload=441449]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:31:36.994045901 (503h2m36.865128966s) [upload=441449]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:31:37.298152335 (503h2m37.16923537s) [upload=441449]: Done uploading segments...
Dec 26 14:31:37.298171903 (503h2m37.169254938s) [upload=441449]: Upload finished. err: <nil>
Duration: 5.77116358s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
19 out of 60 is failed (threshold=3) [X.XX.....X..........X..X..X.X.X....X.....XXX.XX.XX.....X...X] duration:
1m0.003391318s trace:
https://jaeger.dev.storj.io/trace/7cfa2780c86fd364
FAIL (2025-12-26 14:32:05 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: ODNKMB5XWYYNM3FEXNZ4KWFJO7CTJCLLLS5ASLKN4B2CQPX34MPA) context deadline exceeded; piecestore: (Node ID: 1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, Piece ID: NIYLHM3KF5TWQPY5MMIRI2AT3SIC7PSBVAGKHNGHZAYIUZICVQRQ) context deadline exceeded; piecestore: (Node ID: 1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, Piece ID: GFPJBB7JAAX6XED6EUF63WFFUQSOSUVKPLN22ZIE53XDSB6ATDXA) context deadline exceeded; piecestore: (Node ID: 12eshfvC8guQXYqSeUqwmycyBbM4ens9W98cktmW7nPzk1D6pwy, Piece ID: OZ6R7LVR3ZSZEYNZN2X7377VE7LUV4NCVGZHQ2EP7674JTMN76VQ) context deadline exceeded
======LOGS======
seg_pos={0 0}, upload=441442]: Uploading piece...
Dec 26 14:31:06.136083657 (503h2m6.007166692s) [noise=true , address=45.91.116.8:4063 , piece_id=55ITF2D4X2FE3VXIVBVLM2H4EZHPS7BHXMJDDB3D5T2E7PZNUZPQ, seg_pos={0 0}, upload=441442]: Uploading piece...
Dec 26 14:31:06.136417762 (503h2m6.007500797s) [noise=true , address=145.79.227.27:4701 , piece_id=RKQ4O6PZKSXJ4N4D7DSWRRVP4PQ4RY5DXVPJOSBSM72PLACWUWKA, seg_pos={0 0}, upload=441442]: Uploading piece...
Dec 26 14:31:06.136718195 (503h2m6.00780123s) [noise=true , address=77.90.31.19:4863 , piece_id=56MQ4P7U3HMSBVYCFSSHXCRXZ6HJY6LG7GAHX7VRWVADLBX7HOXA, seg_pos={0 0}, upload=441442]: Uploading piece...
Dec 26 14:31:06.136737302 (503h2m6.007820347s) [noise=true , address=145.79.227.30:4773 , piece_id=XEUQR6QP5JBN7N4KWWA6JAAZCIZ3OAONFXFVZMVNWEGXETVMZVGQ, seg_pos={0 0}, upload=441442]: Uploading piece...
Dec 26 14:31:06.137049075 (503h2m6.00813212s) [noise=true , address=145.79.227.49:4367 , piece_id=F77NKUSYSA5IX3RH4GAN7ZGOLKRC56SBGUVY36TSYPMFPFSENQFQ, seg_pos={0 0}, upload=441442]: Uploading piece...
Dec 26 14:31:06.137383131 (503h2m6.008466176s) [noise=true , address=64.49.13.154:4930 , piece_id=M7XPBQJSFGJDJLGNB6TWRVT3CWKSN3IKEHPE7QAHEPQ6VVVBR6MA, seg_pos={0 0}, upload=441442]: Uploading piece...
Dec 26 14:31:06.137467432 (503h2m6.008550467s) [noise=true , address=77.90.31.15:4732 , piece_id=F64S4GAY4USR7ACOXHON54XCPK2DLNZP5U3FXNQIRDDD7PWW444A, seg_pos={0 0}, upload=441442]: Uploading piece...
Dec 26 14:31:06.137860670 (503h2m6.008943705s) [noise=true , address=64.49.13.137:4502 , piece_id=UHVS6MBR6YJCLORZVBOT6LFIYDAE6SMOMJ7NZYZBIYD4GS5GEH2Q, seg_pos={0 0}, upload=441442]: Uploading piece...
Dec 26 14:31:06.137970239 (503h2m6.009053284s) [noise=true , address=77.90.31.13:4631 , piece_id=GFPJBB7JAAX6XED6EUF63WFFUQSOSUVKPLN22ZIE53XDSB6ATDXA, seg_pos={0 0}, upload=441442]: Uploading piece...
Dec 26 14:31:06.138288445 (503h2m6.00937149s) [noise=true , address=145.79.227.44:4200 , piece_id=ANMZFABFW2CJKONIEVARFCBTE7X352ZI52BPIVDF2652EE4GDKUQ, seg_pos={0 0}, upload=441442]: Uploading piece...
Dec 26 14:31:06.138495779 (503h2m6.009578825s) [noise=true , address=77.90.31.14:4673 , piece_id=ODNKMB5XWYYNM3FEXNZ4KWFJO7CTJCLLLS5ASLKN4B2CQPX34MPA, seg_pos={0 0}, upload=441442]: Uploading piece...
Dec 26 14:31:06.138614585 (503h2m6.00969763s) [noise=true , address=145.79.227.14:4855 , piece_id=SRBIQJTKPGDG65JYPVM7MZZ6YM2QSEJ3YZUMLZUHKMA6B7NCZILQ, seg_pos={0 0}, upload=441442]: Uploading piece...
Dec 26 14:31:06.138944394 (503h2m6.010027439s) [noise=true , address=145.79.227.34:4934 , piece_id=SO7TC2RN5ZHJCYC5KXUQPIRI4BPOFKYORV765K2OXFAMRRU7OO7Q, seg_pos={0 0}, upload=441442]: Uploading piece...
Dec 26 14:31:06.138965344 (503h2m6.010048389s) [noise=true , address=64.49.13.169:4715 , piece_id=4TKGTG2GXY7MUCFUU7XEAUPBSNQUHJHUHEL3NKBRYBOXYVI2YUNQ, seg_pos={0 0}, upload=441442]: Uploading piece...
Dec 26 14:31:06.139317835 (503h2m6.01040088s) [noise=true , address=77.90.31.16:4756 , piece_id=NIYLHM3KF5TWQPY5MMIRI2AT3SIC7PSBVAGKHNGHZAYIUZICVQRQ, seg_pos={0 0}, upload=441442]: Uploading piece...
Dec 26 14:31:06.139697869 (503h2m6.010780904s) [noise=true , address=145.79.227.33:4881 , piece_id=6YR7A24VTTO3BZ3SMZDIL6MQMHWTV6YWBNOTBQCB5M6H4JKPILKA, seg_pos={0 0}, upload=441442]: Uploading piece...
Dec 26 14:31:07.439375070 (503h2m7.310458115s) [upload=441442]: Next returned nil segment
Dec 26 14:31:07.439393025 (503h2m7.31047607s) [upload=441442]: Waiting for error group managing segments...
Dec 26 14:31:07.763699935 (503h2m7.63478298s) [noise=true , address=64.49.13.154:4930 , piece_id=M7XPBQJSFGJDJLGNB6TWRVT3CWKSN3IKEHPE7QAHEPQ6VVVBR6MA, seg_pos={0 0}, upload=441442]: Done uploading piece. err: <nil>
Dec 26 14:31:07.844222994 (503h2m7.71530604s) [noise=true , address=64.49.13.137:4502 , piece_id=UHVS6MBR6YJCLORZVBOT6LFIYDAE6SMOMJ7NZYZBIYD4GS5GEH2Q, seg_pos={0 0}, upload=441442]: Done uploading piece. err: <nil>
Dec 26 14:31:08.010303543 (503h2m7.881386588s) [noise=true , address=145.79.227.33:4881 , piece_id=6YR7A24VTTO3BZ3SMZDIL6MQMHWTV6YWBNOTBQCB5M6H4JKPILKA, seg_pos={0 0}, upload=441442]: Done uploading piece. err: <nil>
Dec 26 14:31:08.010517630 (503h2m7.881600665s) [noise=true , address=145.79.227.27:4701 , piece_id=RKQ4O6PZKSXJ4N4D7DSWRRVP4PQ4RY5DXVPJOSBSM72PLACWUWKA, seg_pos={0 0}, upload=441442]: Done uploading piece. err: <nil>
Dec 26 14:31:08.018196512 (503h2m7.889279547s) [noise=true , address=145.79.227.14:4855 , piece_id=SRBIQJTKPGDG65JYPVM7MZZ6YM2QSEJ3YZUMLZUHKMA6B7NCZILQ, seg_pos={0 0}, upload=441442]: Done uploading piece. err: <nil>
Dec 26 14:31:08.030614809 (503h2m7.901697844s) [noise=true , address=145.79.227.49:4367 , piece_id=F77NKUSYSA5IX3RH4GAN7ZGOLKRC56SBGUVY36TSYPMFPFSENQFQ, seg_pos={0 0}, upload=441442]: Done uploading piece. err: <nil>
Dec 26 14:31:08.317495983 (503h2m8.188579018s) [noise=true , address=77.90.31.15:4732 , piece_id=F64S4GAY4USR7ACOXHON54XCPK2DLNZP5U3FXNQIRDDD7PWW444A, seg_pos={0 0}, upload=441442]: Done uploading piece. err: <nil>
Dec 26 14:31:08.320322996 (503h2m8.191406041s) [noise=true , address=77.90.31.13:4631 , piece_id=GFPJBB7JAAX6XED6EUF63WFFUQSOSUVKPLN22ZIE53XDSB6ATDXA, seg_pos={0 0}, upload=441442]: Done uploading piece. err: <nil>
Dec 26 14:31:08.328543701 (503h2m8.199626746s) [noise=true , address=77.90.31.14:4673 , piece_id=ODNKMB5XWYYNM3FEXNZ4KWFJO7CTJCLLLS5ASLKN4B2CQPX34MPA, seg_pos={0 0}, upload=441442]: Done uploading piece. err: <nil>
Dec 26 14:31:08.442643076 (503h2m8.313726121s) [noise=true , address=77.90.31.11:4571 , piece_id=OZ6R7LVR3ZSZEYNZN2X7377VE7LUV4NCVGZHQ2EP7674JTMN76VQ, seg_pos={0 0}, upload=441442]: Done uploading piece. err: <nil>
Dec 26 14:31:08.732094244 (503h2m8.603177279s) [noise=true , address=45.91.116.8:4063 , piece_id=55ITF2D4X2FE3VXIVBVLM2H4EZHPS7BHXMJDDB3D5T2E7PZNUZPQ, seg_pos={0 0}, upload=441442]: Done uploading piece. err: <nil>
Dec 26 14:31:08.890501049 (503h2m8.761584084s) [noise=true , address=77.90.31.16:4756 , piece_id=NIYLHM3KF5TWQPY5MMIRI2AT3SIC7PSBVAGKHNGHZAYIUZICVQRQ, seg_pos={0 0}, upload=441442]: Done uploading piece. err: <nil>
Dec 26 14:31:08.933994509 (503h2m8.805077554s) [noise=true , address=145.79.227.44:4200 , piece_id=ANMZFABFW2CJKONIEVARFCBTE7X352ZI52BPIVDF2652EE4GDKUQ, seg_pos={0 0}, upload=441442]: Done uploading piece. err: <nil>
Dec 26 14:31:09.032189019 (503h2m8.903272065s) [noise=true , address=145.79.227.30:4773 , piece_id=XEUQR6QP5JBN7N4KWWA6JAAZCIZ3OAONFXFVZMVNWEGXETVMZVGQ, seg_pos={0 0}, upload=441442]: Done uploading piece. err: <nil>
Dec 26 14:31:09.032207694 (503h2m8.90329073s) [seg_pos={0 0}, upload=441442]: Segment reached optimal threshold of 14 pieces.
Dec 26 14:31:09.032429967 (503h2m8.903513012s) [noise=true , address=77.90.31.19:4863 , piece_id=56MQ4P7U3HMSBVYCFSSHXCRXZ6HJY6LG7GAHX7VRWVADLBX7HOXA, seg_pos={0 0}, upload=441442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL): protocol: expected piece hash; context canceled; EOF
Dec 26 14:31:09.032438745 (503h2m8.90352178s) [noise=true , address=145.79.227.40:4066 , piece_id=76OKIXWJ54AAFEX7DXUDCYVOE7BRZ35WDL2CANXS7XM2XF2B46EQ, seg_pos={0 0}, upload=441442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF
Dec 26 14:31:09.032448934 (503h2m8.903531979s) [noise=true , address=145.79.227.40:4066 , piece_id=76OKIXWJ54AAFEX7DXUDCYVOE7BRZ35WDL2CANXS7XM2XF2B46EQ, seg_pos={0 0}, upload=441442]: Long tail cancellation due to optimal threshold
Dec 26 14:31:09.032458181 (503h2m8.903541216s) [noise=true , address=77.90.31.19:4863 , piece_id=56MQ4P7U3HMSBVYCFSSHXCRXZ6HJY6LG7GAHX7VRWVADLBX7HOXA, seg_pos={0 0}, upload=441442]: Long tail cancellation due to optimal threshold
Dec 26 14:31:09.032587939 (503h2m8.903670984s) [noise=true , address=64.49.13.169:4715 , piece_id=4TKGTG2GXY7MUCFUU7XEAUPBSNQUHJHUHEL3NKBRYBOXYVI2YUNQ, seg_pos={0 0}, upload=441442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:37698->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:37698->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37698->64.49.13.169:4715: use of closed network connection
Dec 26 14:31:09.032596124 (503h2m8.903679169s) [noise=true , address=64.49.13.169:4715 , piece_id=4TKGTG2GXY7MUCFUU7XEAUPBSNQUHJHUHEL3NKBRYBOXYVI2YUNQ, seg_pos={0 0}, upload=441442]: Long tail cancellation due to optimal threshold
Dec 26 14:31:09.032779865 (503h2m8.9038629s) [noise=true , address=145.79.227.34:4934 , piece_id=SO7TC2RN5ZHJCYC5KXUQPIRI4BPOFKYORV765K2OXFAMRRU7OO7Q, seg_pos={0 0}, upload=441442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TpurbQEmcBR8qoHKmAtxJAQinkzi8p5tX4Bn5i3UbDQ3kZHGT): protocol: expected piece hash; context canceled; EOF
Dec 26 14:31:09.032787128 (503h2m8.903870163s) [noise=true , address=145.79.227.34:4934 , piece_id=SO7TC2RN5ZHJCYC5KXUQPIRI4BPOFKYORV765K2OXFAMRRU7OO7Q, seg_pos={0 0}, upload=441442]: Long tail cancellation due to optimal threshold
Dec 26 14:31:09.032804100 (503h2m8.903887135s) [seg_pos={0 0}, upload=441442]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 14:31:09.032885115 (503h2m8.90396816s) [upload=441442]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:31:09.032889784 (503h2m8.903972819s) [upload=441442]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:31:09.331015740 (503h2m9.202098785s) [upload=441442]: Done uploading segments...
Dec 26 14:31:09.331035959 (503h2m9.202118994s) [upload=441442]: Upload finished. err: <nil>
Duration: 1m0.003406336s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
13.548963402s trace:
https://jaeger.dev.storj.io/trace/3ccdb98964ad100c
FAIL (2025-12-26 14:32:05 UTC)
orbiter warning: context deadline exceeded
Duration: 13.548986607s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [XX..XX...X...X...........X..X.X...........X....XX.XXX..XXX.X] duration:
30.003717061s trace:
https://jaeger.dev.storj.io/trace/4957367a1f492fcb
FAIL (2025-12-26 14:32:02 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 14:31:33.597437556 (503h2m33.468520601s) [upload=441449]: Starting upload
Dec 26 14:31:33.597553948 (503h2m33.468636993s) [upload=441449]: Done starting upload
Dec 26 14:31:33.597603222 (503h2m33.468686267s) [upload=441449]: Uploading segments...
Dec 26 14:31:33.597789576 (503h2m33.468872611s) [upload=441449]: Got next segment. Inline: false
Dec 26 14:31:33.597882563 (503h2m33.468965598s) [upload=441449]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 14:31:33.597889186 (503h2m33.468972231s) [upload=441449]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 14:31:33.846933082 (503h2m33.718016127s) [seg_pos={0 0}, upload=441449]: Begin upload segment...
Dec 26 14:31:33.846983659 (503h2m33.718066694s) [seg_pos={0 0}, upload=441449]: Done begin upload segment.
Dec 26 14:31:33.847017223 (503h2m33.718100268s) [noise=true , address=145.79.227.51:4409 , piece_id=DL3T4T3DOTHIPXAZ5BEEM42OCD3FKTU5XOHVWNDIUQE3ERZFLWZQ, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.847050697 (503h2m33.718133732s) [noise=true , address=77.90.31.14:4661 , piece_id=YFYXETEUCFVK7MBM3QCVVSE4FR4SHJBHRZLD4WHTI64RDSGMGQNQ, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.847570256 (503h2m33.718653311s) [noise=true , address=77.90.31.17:4773 , piece_id=FHH435DDQIZKXMHG76N47EK7JD5XC3IOXFUSSFPDFMKBOLBN2OKA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.848112698 (503h2m33.719195744s) [noise=true , address=45.91.116.4:4940 , piece_id=5X6DK3LI6ASO6EYUS6UTWNZDOR7U3IMVBBY6M73IEAZ52RWS2HYA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.848499495 (503h2m33.71958254s) [noise=true , address=145.79.227.49:4337 , piece_id=JVMK4NXGKUWEP2XAAZZWTSE5TEPFYVGKE3FFWDMOVHRK6Z27K4RA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.848875521 (503h2m33.719958556s) [noise=true , address=145.79.227.45:4229 , piece_id=5RKLGUD2ZO5JEAEK3GVEKTKBHYDVOQ5BK5NYJZQUB7P2I5NBFDBA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.849364823 (503h2m33.720447868s) [noise=true , address=145.79.227.21:4473 , piece_id=UCQK72VUEHSCCJBIEZJ4UD7SMFZUPXK4KM6VGB47RPMOOD2HVTHQ, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.849763733 (503h2m33.720846768s) [noise=true , address=45.91.116.14:4190 , piece_id=ZGP23KORYXHV2ZNPQSDTLCR7PR4HCYPVJ7QCTA6DBYHRSNJAGXPA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.850922750 (503h2m33.722005795s) [noise=true , address=145.79.227.36:4518 , piece_id=LJXTQWBSIVHP5JDCT6PEWOICJFW72MGSRPBOYDIYCJEKRRMLGOLA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.851242398 (503h2m33.722325443s) [noise=true , address=45.91.116.5:4298 , piece_id=6AY2UYXYETFVFKWHNY7OHUONOYAMA26N2YF3RHAYFA2ZZW2ZNQ3Q, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.852200212 (503h2m33.723283247s) [noise=true , address=145.79.227.47:4266 , piece_id=VRNGVBKFWQRJA7TAR6QZ52JVFLKTCNBOYK7VSZSUUJVPFAJBA6GA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.853435684 (503h2m33.72451873s) [noise=true , address=145.79.227.38:4979 , piece_id=52FH5WDZ76TN3LRN6IMMTS5OE3TXC637YY53IAEU7MHGHRE7R7JA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.854620921 (503h2m33.725703956s) [noise=true , address=145.79.227.50:4375 , piece_id=23RC4QNCFLTCIF4JHVD7ESHDIDW2CWWKCSE5Z64YX3P5526GDCJA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.856049742 (503h2m33.727132787s) [noise=true , address=77.90.31.19:4844 , piece_id=O56QQ4AUPE24ERTZQYUQEOK6NYNWYI3DNZQ5FWKFEZLLHF63OANQ, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.857977794 (503h2m33.729060839s) [noise=true , address=77.90.31.11:4555 , piece_id=W7NU2VVWGDDND474EICGZYS3GPOR75FNBD7Q6ZVIZVEQ2CRYCNKQ, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:33.858982477 (503h2m33.730065512s) [noise=true , address=145.79.227.29:4943 , piece_id=Y7JPU43H23ZITBIQ6TK7RUZPG3T3NMGPDTB7S6DJNZG7IVGTUYVA, seg_pos={0 0}, upload=441449]: Uploading piece...
Dec 26 14:31:35.571477638 (503h2m35.442560673s) [upload=441449]: Next returned nil segment
Dec 26 14:31:35.571490583 (503h2m35.442573628s) [upload=441449]: Waiting for error group managing segments...
Dec 26 14:31:35.855026257 (503h2m35.726109302s) [noise=true , address=145.79.227.49:4337 , piece_id=JVMK4NXGKUWEP2XAAZZWTSE5TEPFYVGKE3FFWDMOVHRK6Z27K4RA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:35.862095689 (503h2m35.733178734s) [noise=true , address=145.79.227.38:4979 , piece_id=52FH5WDZ76TN3LRN6IMMTS5OE3TXC637YY53IAEU7MHGHRE7R7JA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:35.862796403 (503h2m35.733879438s) [noise=true , address=145.79.227.36:4518 , piece_id=LJXTQWBSIVHP5JDCT6PEWOICJFW72MGSRPBOYDIYCJEKRRMLGOLA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:35.864289396 (503h2m35.735372431s) [noise=true , address=145.79.227.50:4375 , piece_id=23RC4QNCFLTCIF4JHVD7ESHDIDW2CWWKCSE5Z64YX3P5526GDCJA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:35.869609206 (503h2m35.740692251s) [noise=true , address=145.79.227.45:4229 , piece_id=5RKLGUD2ZO5JEAEK3GVEKTKBHYDVOQ5BK5NYJZQUB7P2I5NBFDBA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:35.872899100 (503h2m35.743982135s) [noise=true , address=145.79.227.21:4473 , piece_id=UCQK72VUEHSCCJBIEZJ4UD7SMFZUPXK4KM6VGB47RPMOOD2HVTHQ, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:35.881408674 (503h2m35.752491709s) [noise=true , address=145.79.227.29:4943 , piece_id=Y7JPU43H23ZITBIQ6TK7RUZPG3T3NMGPDTB7S6DJNZG7IVGTUYVA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:35.888713384 (503h2m35.759796429s) [noise=true , address=145.79.227.51:4409 , piece_id=DL3T4T3DOTHIPXAZ5BEEM42OCD3FKTU5XOHVWNDIUQE3ERZFLWZQ, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:35.893099275 (503h2m35.76418231s) [noise=true , address=145.79.227.47:4266 , piece_id=VRNGVBKFWQRJA7TAR6QZ52JVFLKTCNBOYK7VSZSUUJVPFAJBA6GA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:36.043138096 (503h2m35.914221141s) [noise=true , address=77.90.31.14:4661 , piece_id=YFYXETEUCFVK7MBM3QCVVSE4FR4SHJBHRZLD4WHTI64RDSGMGQNQ, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:36.087256205 (503h2m35.95833924s) [noise=true , address=45.91.116.5:4298 , piece_id=6AY2UYXYETFVFKWHNY7OHUONOYAMA26N2YF3RHAYFA2ZZW2ZNQ3Q, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:36.233410791 (503h2m36.104493836s) [noise=true , address=45.91.116.14:4190 , piece_id=ZGP23KORYXHV2ZNPQSDTLCR7PR4HCYPVJ7QCTA6DBYHRSNJAGXPA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:36.993343372 (503h2m36.864426407s) [noise=true , address=77.90.31.19:4844 , piece_id=O56QQ4AUPE24ERTZQYUQEOK6NYNWYI3DNZQ5FWKFEZLLHF63OANQ, seg_pos={0 0}, upload=441449]: Done uploading piece. err: <nil>
Dec 26 14:31:36.993368200 (503h2m36.864451285s) [seg_pos={0 0}, upload=441449]: Segment reached optimal threshold of 13 pieces.
Dec 26 14:31:36.993595764 (503h2m36.864678799s) [noise=true , address=77.90.31.11:4555 , piece_id=W7NU2VVWGDDND474EICGZYS3GPOR75FNBD7Q6ZVIZVEQ2CRYCNKQ, seg_pos={0 0}, upload=441449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: write tcp 188.166.225.170:46760->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:46760->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46760->77.90.31.11:4555: use of closed network connection
Dec 26 14:31:36.993605442 (503h2m36.864688477s) [noise=true , address=77.90.31.11:4555 , piece_id=W7NU2VVWGDDND474EICGZYS3GPOR75FNBD7Q6ZVIZVEQ2CRYCNKQ, seg_pos={0 0}, upload=441449]: Long tail cancellation due to optimal threshold
Dec 26 14:31:36.993751880 (503h2m36.864834946s) [noise=true , address=77.90.31.17:4773 , piece_id=FHH435DDQIZKXMHG76N47EK7JD5XC3IOXFUSSFPDFMKBOLBN2OKA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF
Dec 26 14:31:36.993770246 (503h2m36.864853291s) [noise=true , address=77.90.31.17:4773 , piece_id=FHH435DDQIZKXMHG76N47EK7JD5XC3IOXFUSSFPDFMKBOLBN2OKA, seg_pos={0 0}, upload=441449]: Long tail cancellation due to optimal threshold
Dec 26 14:31:36.993934509 (503h2m36.865017544s) [noise=true , address=45.91.116.4:4940 , piece_id=5X6DK3LI6ASO6EYUS6UTWNZDOR7U3IMVBBY6M73IEAZ52RWS2HYA, seg_pos={0 0}, upload=441449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF
Dec 26 14:31:36.993942774 (503h2m36.865025809s) [noise=true , address=45.91.116.4:4940 , piece_id=5X6DK3LI6ASO6EYUS6UTWNZDOR7U3IMVBBY6M73IEAZ52RWS2HYA, seg_pos={0 0}, upload=441449]: Long tail cancellation due to optimal threshold
Dec 26 14:31:36.993961339 (503h2m36.865044384s) [seg_pos={0 0}, upload=441449]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 14:31:36.994038988 (503h2m36.865122033s) [upload=441449]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:31:36.994045901 (503h2m36.865128966s) [upload=441449]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:31:37.298152335 (503h2m37.16923537s) [upload=441449]: Done uploading segments...
Dec 26 14:31:37.298171903 (503h2m37.169254938s) [upload=441449]: Upload finished. err: <nil>
Duration: 30.003739775s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
54 out of 60 is failed (threshold=6) [XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX.XXXXX.XXXXXXX.XXXXXXXXX.] duration:
4.645672048s trace:
https://jaeger.dev.storj.io/trace/3cf14fba16b63f3c
FAIL (2025-12-26 14:32:00 UTC)
orbiter warning: context deadline exceeded
Duration: 4.645700142s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
14.874748118s trace:
https://jaeger.dev.storj.io/trace/5b63846e68ea1b18
FAIL (2025-12-26 14:31:53 UTC)
orbiter warning: context deadline exceeded
Duration: 14.874778125s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
54 out of 60 is failed (threshold=6) [.XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX.XXXXX.XXXXXXX.XXXXXXXXX] duration:
5.000820408s trace:
https://jaeger.dev.storj.io/trace/3cf14fba16b63f3c
FAIL (2025-12-26 14:31:22 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000836478s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
18 out of 60 is failed (threshold=3) [.X.XX.....X..........X..X..X.X.X....X.....XXX.XX.XX.....X...] duration:
52.028429973s trace:
https://jaeger.dev.storj.io/trace/3b3dc37aa1526738
FAIL (2025-12-26 14:31:04 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1nstPr2GMJo6pnBmBuDME5QYw3mXgnqLoaTKkcKUhjSbAY8mQk, Piece ID: UAASM7Q6N6SJFL3BZQG4J3PWGICDNX7HWKKMLAE3Z4DTLF2CQAHQ) context deadline exceeded; piecestore: (Node ID: 12HcwSEGPEpd59dUukENRoUykRyqmVLiGcKEBbJ72qr9b5tpoEY, Piece ID: WSRZPIBVTNRPGFTBESN6ZEWLITLZOZLBB4IPRAUPYH3M6YEQOWZQ) context deadline exceeded; piecestore: (Node ID: 12QnZ3q9oe4zPuLpS3V1DCBhyaU1umS5B7bpuAqPkaaY96ErWbo, Piece ID: XZSYVBXPVPA73GHQBP7PFICVBSO3RNC3YLIOSYGIPBKQ66FHQPWA) context deadline exceeded; piecestore: (Node ID: 121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, Piece ID: 7GZ6PRPYTSWSCFWRSFIBJ5EKN7FETMBWUYJKSS2YVKG3JJK42S3A) context deadline exceeded; piecestore: (Node ID: 1XjJpLBEZm5daHcBUmUyC7QpLVhu9Vm9ex9RYcYJWEuLGk527F, Piece ID: J2UOIRDKCGLOS6FPEQCM5SZWGPDPRPO6O2HFY2KJMSWTFZEPRAMQ) context deadline exceeded
======LOGS======
7s) [noise=true , address=45.91.116.12:4236 , piece_id=M5R7IAUXCFF7QF4HHWME6H7PKQJYRKBGLGU5VFJJFQE6LUPIYCJA, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:12.729380403 (502h58m12.600463448s) [noise=true , address=77.90.31.21:4909 , piece_id=7GZ6PRPYTSWSCFWRSFIBJ5EKN7FETMBWUYJKSS2YVKG3JJK42S3A, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:12.729753183 (502h58m12.600836228s) [noise=true , address=64.49.13.139:4480 , piece_id=KGAH4B57THDLG3FIMSEKYO2PE5JXKYLBLRMQQH37CKR5YCBL3QGQ, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:12.730155719 (502h58m12.601238764s) [noise=true , address=45.91.116.10:4097 , piece_id=WSRZPIBVTNRPGFTBESN6ZEWLITLZOZLBB4IPRAUPYH3M6YEQOWZQ, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:12.730540132 (502h58m12.601623217s) [noise=true , address=145.79.227.20:4452 , piece_id=UGIUMXQNQIXGGNF2BOQTADA5JOUQKEI27DTWXFB6DIJUJ2GTGBQQ, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:12.730922941 (502h58m12.602005986s) [noise=true , address=145.79.227.40:4076 , piece_id=XVEBWXMTJ7VK2KCPYXS2LDXP2BRI2JYMV6EZF6XMMVTMQ4ICA2BQ, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:12.731266324 (502h58m12.602349369s) [noise=true , address=145.79.227.34:4934 , piece_id=ATCG3GH3MMQSVE5AN6H4JD7WKXPFAOXIR2YIHTHXY2AC2OKPWBMQ, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:12.731693027 (502h58m12.602776062s) [noise=true , address=77.90.31.17:4799 , piece_id=NG3PPYMUUODKZVAVYKGTWZ2QBHOHINP45ZAMOBTQBZBJ6VOPPFNA, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:12.732138174 (502h58m12.603221209s) [noise=true , address=77.90.31.13:4640 , piece_id=J2UOIRDKCGLOS6FPEQCM5SZWGPDPRPO6O2HFY2KJMSWTFZEPRAMQ, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:12.732479955 (502h58m12.603563s) [noise=true , address=64.49.13.27:4957 , piece_id=SI7ANTUA53SPUXSLY2ECQUDHIHWHPHMFTPFWNBMANKZGUW4U2L4A, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:12.732773655 (502h58m12.60385669s) [noise=true , address=77.90.31.18:4832 , piece_id=XZSYVBXPVPA73GHQBP7PFICVBSO3RNC3YLIOSYGIPBKQ66FHQPWA, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:12.733031345 (502h58m12.60411439s) [noise=true , address=64.49.13.169:4711 , piece_id=Q2LQKJ2RBKW2QLHGWIVPWJKBCUQZBQFFOPJGEWMPZZWAILDUJBUQ, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:12.733433241 (502h58m12.604516286s) [noise=true , address=77.90.31.12:4614 , piece_id=5CPNE3UG2AWMOSC2MA6VOSJQLUUDENYYBUQGFHXYN3ON63PVM7EQ, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:12.733723934 (502h58m12.604806969s) [noise=true , address=77.90.31.16:4753 , piece_id=2GMK4UAMU6AHDTQEW447NXYGDKKSZOUFJ7PGL5OGJ5W2GIECTQMQ, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:12.744912929 (502h58m12.615995964s) [noise=true , address=145.79.227.33:4884 , piece_id=JDO3F6HQRBGHWJ74ZVJNML2X5757PP55DWQLF4YT3P7JDKZ7VZIA, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:12.748930370 (502h58m12.620013415s) [noise=true , address=145.79.227.30:4772 , piece_id=MBZYVHFEAHN2FEVS6LHPUCCBHX74E35JO6EOG4GPKBRG7UIK2IPA, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:14.090029017 (502h58m13.961112062s) [upload=441384]: Next returned nil segment
Dec 26 14:27:14.090042262 (502h58m13.961125297s) [upload=441384]: Waiting for error group managing segments...
Dec 26 14:27:14.444533658 (502h58m14.315616693s) [noise=true , address=64.49.13.139:4480 , piece_id=KGAH4B57THDLG3FIMSEKYO2PE5JXKYLBLRMQQH37CKR5YCBL3QGQ, seg_pos={0 0}, upload=441384]: Done uploading piece. err: <nil>
Dec 26 14:27:14.586227048 (502h58m14.457310083s) [noise=true , address=64.49.13.27:4957 , piece_id=SI7ANTUA53SPUXSLY2ECQUDHIHWHPHMFTPFWNBMANKZGUW4U2L4A, seg_pos={0 0}, upload=441384]: Done uploading piece. err: <nil>
Dec 26 14:27:14.622921881 (502h58m14.494004916s) [noise=true , address=145.79.227.20:4452 , piece_id=UGIUMXQNQIXGGNF2BOQTADA5JOUQKEI27DTWXFB6DIJUJ2GTGBQQ, seg_pos={0 0}, upload=441384]: Done uploading piece. err: <nil>
Dec 26 14:27:14.812511214 (502h58m14.683594259s) [noise=true , address=77.90.31.21:4909 , piece_id=7GZ6PRPYTSWSCFWRSFIBJ5EKN7FETMBWUYJKSS2YVKG3JJK42S3A, seg_pos={0 0}, upload=441384]: Done uploading piece. err: <nil>
Dec 26 14:27:14.823469320 (502h58m14.694552355s) [noise=true , address=77.90.31.18:4832 , piece_id=XZSYVBXPVPA73GHQBP7PFICVBSO3RNC3YLIOSYGIPBKQ66FHQPWA, seg_pos={0 0}, upload=441384]: Done uploading piece. err: <nil>
Dec 26 14:27:14.884791981 (502h58m14.755875016s) [noise=true , address=77.90.31.15:4725 , piece_id=UAASM7Q6N6SJFL3BZQG4J3PWGICDNX7HWKKMLAE3Z4DTLF2CQAHQ, seg_pos={0 0}, upload=441384]: Done uploading piece. err: <nil>
Dec 26 14:27:14.897429475 (502h58m14.76851251s) [noise=true , address=77.90.31.12:4614 , piece_id=5CPNE3UG2AWMOSC2MA6VOSJQLUUDENYYBUQGFHXYN3ON63PVM7EQ, seg_pos={0 0}, upload=441384]: Done uploading piece. err: <nil>
Dec 26 14:27:14.921617022 (502h58m14.792700057s) [noise=true , address=77.90.31.13:4640 , piece_id=J2UOIRDKCGLOS6FPEQCM5SZWGPDPRPO6O2HFY2KJMSWTFZEPRAMQ, seg_pos={0 0}, upload=441384]: Done uploading piece. err: <nil>
Dec 26 14:27:15.217935085 (502h58m15.089018121s) [noise=true , address=77.90.31.17:4799 , piece_id=NG3PPYMUUODKZVAVYKGTWZ2QBHOHINP45ZAMOBTQBZBJ6VOPPFNA, seg_pos={0 0}, upload=441384]: Done uploading piece. err: <nil>
Dec 26 14:27:15.538396774 (502h58m15.409479819s) [noise=true , address=45.91.116.10:4097 , piece_id=WSRZPIBVTNRPGFTBESN6ZEWLITLZOZLBB4IPRAUPYH3M6YEQOWZQ, seg_pos={0 0}, upload=441384]: Done uploading piece. err: <nil>
Dec 26 14:27:15.575796018 (502h58m15.446879063s) [noise=true , address=145.79.227.34:4934 , piece_id=ATCG3GH3MMQSVE5AN6H4JD7WKXPFAOXIR2YIHTHXY2AC2OKPWBMQ, seg_pos={0 0}, upload=441384]: Done uploading piece. err: <nil>
Dec 26 14:27:15.588172846 (502h58m15.459255881s) [noise=true , address=145.79.227.30:4772 , piece_id=MBZYVHFEAHN2FEVS6LHPUCCBHX74E35JO6EOG4GPKBRG7UIK2IPA, seg_pos={0 0}, upload=441384]: Done uploading piece. err: <nil>
Dec 26 14:27:15.654034369 (502h58m15.525117404s) [noise=true , address=145.79.227.44:4221 , piece_id=KNE5IJVSCFLIOLLXHC32A4X4DMVQKV6WHWKA7NSWSYYC4F7OMOFA, seg_pos={0 0}, upload=441384]: Done uploading piece. err: <nil>
Dec 26 14:27:15.764565211 (502h58m15.635648246s) [noise=true , address=145.79.227.40:4076 , piece_id=XVEBWXMTJ7VK2KCPYXS2LDXP2BRI2JYMV6EZF6XMMVTMQ4ICA2BQ, seg_pos={0 0}, upload=441384]: Done uploading piece. err: <nil>
Dec 26 14:27:15.764584317 (502h58m15.635667352s) [seg_pos={0 0}, upload=441384]: Segment reached optimal threshold of 14 pieces.
Dec 26 14:27:15.764749782 (502h58m15.635832817s) [noise=true , address=64.49.13.169:4711 , piece_id=Q2LQKJ2RBKW2QLHGWIVPWJKBCUQZBQFFOPJGEWMPZZWAILDUJBUQ, seg_pos={0 0}, upload=441384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:36534->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:36534->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36534->64.49.13.169:4711: use of closed network connection
Dec 26 14:27:15.764755412 (502h58m15.635838447s) [noise=true , address=64.49.13.169:4711 , piece_id=Q2LQKJ2RBKW2QLHGWIVPWJKBCUQZBQFFOPJGEWMPZZWAILDUJBUQ, seg_pos={0 0}, upload=441384]: Long tail cancellation due to optimal threshold
Dec 26 14:27:15.764867776 (502h58m15.635950811s) [noise=true , address=45.91.116.12:4236 , piece_id=M5R7IAUXCFF7QF4HHWME6H7PKQJYRKBGLGU5VFJJFQE6LUPIYCJA, seg_pos={0 0}, upload=441384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH): protocol: expected piece hash; context canceled; EOF
Dec 26 14:27:15.764897062 (502h58m15.635980097s) [noise=true , address=145.79.227.33:4884 , piece_id=JDO3F6HQRBGHWJ74ZVJNML2X5757PP55DWQLF4YT3P7JDKZ7VZIA, seg_pos={0 0}, upload=441384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qGXdnm83P4eMnCtZdRt2x5yAWP7pQafWwoPG72daZJVR9pyKG): protocol: expected piece hash; context canceled; EOF
Dec 26 14:27:15.764903855 (502h58m15.63598689s) [noise=true , address=145.79.227.33:4884 , piece_id=JDO3F6HQRBGHWJ74ZVJNML2X5757PP55DWQLF4YT3P7JDKZ7VZIA, seg_pos={0 0}, upload=441384]: Long tail cancellation due to optimal threshold
Dec 26 14:27:15.764910517 (502h58m15.635993552s) [noise=true , address=45.91.116.12:4236 , piece_id=M5R7IAUXCFF7QF4HHWME6H7PKQJYRKBGLGU5VFJJFQE6LUPIYCJA, seg_pos={0 0}, upload=441384]: Long tail cancellation due to optimal threshold
Dec 26 14:27:15.765067747 (502h58m15.636150782s) [noise=true , address=77.90.31.16:4753 , piece_id=2GMK4UAMU6AHDTQEW447NXYGDKKSZOUFJ7PGL5OGJ5W2GIECTQMQ, seg_pos={0 0}, upload=441384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE): protocol: expected piece hash; context canceled; EOF
Dec 26 14:27:15.765076293 (502h58m15.636159328s) [noise=true , address=77.90.31.16:4753 , piece_id=2GMK4UAMU6AHDTQEW447NXYGDKKSZOUFJ7PGL5OGJ5W2GIECTQMQ, seg_pos={0 0}, upload=441384]: Long tail cancellation due to optimal threshold
Dec 26 14:27:15.765245906 (502h58m15.636328941s) [seg_pos={0 0}, upload=441384]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 14:27:15.765317863 (502h58m15.636400908s) [upload=441384]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:27:15.765322372 (502h58m15.636405407s) [upload=441384]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:27:16.068012309 (502h58m15.939095344s) [upload=441384]: Done uploading segments...
Dec 26 14:27:16.068037878 (502h58m15.939120913s) [upload=441384]: Upload finished. err: <nil>
Duration: 52.028451324s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [.XX..XX...X...X...........X..X.X...........X....XX.XXX..XXX.] duration:
7.037976353s trace:
https://jaeger.dev.storj.io/trace/591402ad96a55046
FAIL (2025-12-26 14:31:03 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Starting upload
Dec 26 14:30:26.764451839 (503h1m26.635534874s) [upload=441432]: Done starting upload
Dec 26 14:30:26.764500713 (503h1m26.635583748s) [upload=441432]: Uploading segments...
Dec 26 14:30:26.764633014 (503h1m26.635716049s) [upload=441432]: Got next segment. Inline: false
Dec 26 14:30:26.764670596 (503h1m26.635753652s) [upload=441432]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 14:30:26.764676618 (503h1m26.635759663s) [upload=441432]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 14:30:27.009506388 (503h1m26.880589443s) [seg_pos={0 0}, upload=441432]: Begin upload segment...
Dec 26 14:30:27.009571281 (503h1m26.880654327s) [seg_pos={0 0}, upload=441432]: Done begin upload segment.
Dec 26 14:30:27.009613822 (503h1m26.880696867s) [noise=true , address=145.79.227.39:4015 , piece_id=B2EZYPCFXBXODQXFEDKJ5UBANRQRCHY52737EYOMXVT45DAYXZ4Q, seg_pos={0 0}, upload=441432]: Uploading piece...
Dec 26 14:30:27.009634261 (503h1m26.880717306s) [noise=true , address=145.79.227.42:4121 , piece_id=U2JUFVVRCFQ6QUXZONLP4E7GW6E6YYBUF4UBEOA2L22M2TM3MLGA, seg_pos={0 0}, upload=441432]: Uploading piece...
Dec 26 14:30:27.010148059 (503h1m26.881231094s) [noise=true , address=145.79.227.11:4732 , piece_id=G3SYKWBI4P6WET75NQE7CZS6E23RSWVFH2JAYZBXMSW2SBODXAMQ, seg_pos={0 0}, upload=441432]: Uploading piece...
Dec 26 14:30:27.010220778 (503h1m26.881303813s) [noise=true , address=145.79.227.38:4977 , piece_id=ONNO4SRTWCNKXPLXSQLTYI2VZISLTAYTNCKBEMDIXL3P7FWAJJTQ, seg_pos={0 0}, upload=441432]: Uploading piece...
Dec 26 14:30:27.010697746 (503h1m26.881780791s) [noise=true , address=45.91.116.5:4298 , piece_id=VW6YPVYORDEMOD5JU2FKD5CJZGFCSTLW5Y235YOVSUP6JX2CBMRQ, seg_pos={0 0}, upload=441432]: Uploading piece...
Dec 26 14:30:27.011130149 (503h1m26.882213184s) [noise=true , address=145.79.227.47:4266 , piece_id=5YHHH3DYLFYZCIXMJOIH3TJZVLOOPDTVX4Y3P3WPRM4DLBEWFX6A, seg_pos={0 0}, upload=441432]: Uploading piece...
Dec 26 14:30:27.012038069 (503h1m26.883121114s) [noise=true , address=145.79.227.13:4803 , piece_id=EJEGRDHJ4YSB2556QLODTRDGCM2RFDRNJZTC4NKJ4YLFYBQNLWAQ, seg_pos={0 0}, upload=441432]: Uploading piece...
Dec 26 14:30:27.012798265 (503h1m26.883881311s) [noise=true , address=45.91.116.13:4155 , piece_id=URJDHIOTOEQDK5MJVYT5LVXL3AXKLGUJVIK6PLQGRBYYGC3LWMJA, seg_pos={0 0}, upload=441432]: Uploading piece...
Dec 26 14:30:27.013487919 (503h1m26.884570954s) [noise=true , address=45.91.116.7:4009 , piece_id=CZF2WLV3DLMICJMKBLUVCNU4KBB5WSEL5DW2ELLZNODFIYA7XFUQ, seg_pos={0 0}, upload=441432]: Uploading piece...
Dec 26 14:30:27.014309634 (503h1m26.885392679s) [noise=true , address=145.79.227.46:4603 , piece_id=AYJUD7SZI5RL3BVITMSJTXUEEJQY762MMN6Z7W2WZ5VBMWVRCCWA, seg_pos={0 0}, upload=441432]: Uploading piece...
Dec 26 14:30:27.014692703 (503h1m26.885775738s) [noise=true , address=145.79.227.6:4257 , piece_id=OWSV6LYJGXYBOH3QVNU22DZVOFJQLZMZA3KKBIKQKXWHU5LQQZWA, seg_pos={0 0}, upload=441432]: Uploading piece...
Dec 26 14:30:27.015114136 (503h1m26.886197181s) [noise=true , address=145.79.227.16:4910 , piece_id=IYSTF4W4CG2VNOVSN3QZE2EN6DAMGUEFREC6TYHYBLMQ77ZJSAUQ, seg_pos={0 0}, upload=441432]: Uploading piece...
Dec 26 14:30:27.015577458 (503h1m26.886660503s) [noise=true , address=45.91.116.10:4082 , piece_id=M5SLIV4MAQ23GMAAXLG3MOSITXOOXCV3UJ32HXYFYQQACHVDRIOQ, seg_pos={0 0}, upload=441432]: Uploading piece...
Dec 26 14:30:27.016058424 (503h1m26.887141459s) [noise=true , address=145.79.227.21:4475 , piece_id=F7A3TCHUH7GUOBWJBDMLPBU2OJAXP6NAF7T5SCXD4CQS4SDDIBXQ, seg_pos={0 0}, upload=441432]: Uploading piece...
Dec 26 14:30:27.016527687 (503h1m26.887610722s) [noise=true , address=45.91.116.12:4227 , piece_id=OR6YGGDRMUEZLDHU4YWSI2L7UBA5HLDRMIKYKYWWOMAZTUVIJMBQ, seg_pos={0 0}, upload=441432]: Uploading piece...
Dec 26 14:30:27.016858367 (503h1m26.887941402s) [noise=true , address=45.91.116.14:4190 , piece_id=66J7YGDQ7KUVERKEO5AOXCIAGWZSSJAKE7CNBCNMQ2SADH5OJOUQ, seg_pos={0 0}, upload=441432]: Uploading piece...
Dec 26 14:30:28.722398527 (503h1m28.593481562s) [upload=441432]: Next returned nil segment
Dec 26 14:30:28.722412724 (503h1m28.593495759s) [upload=441432]: Waiting for error group managing segments...
Dec 26 14:30:29.011610900 (503h1m28.882693945s) [noise=true , address=145.79.227.39:4015 , piece_id=B2EZYPCFXBXODQXFEDKJ5UBANRQRCHY52737EYOMXVT45DAYXZ4Q, seg_pos={0 0}, upload=441432]: Done uploading piece. err: <nil>
Dec 26 14:30:29.017864138 (503h1m28.888947173s) [noise=true , address=145.79.227.42:4121 , piece_id=U2JUFVVRCFQ6QUXZONLP4E7GW6E6YYBUF4UBEOA2L22M2TM3MLGA, seg_pos={0 0}, upload=441432]: Done uploading piece. err: <nil>
Dec 26 14:30:29.019890516 (503h1m28.890973561s) [noise=true , address=145.79.227.6:4257 , piece_id=OWSV6LYJGXYBOH3QVNU22DZVOFJQLZMZA3KKBIKQKXWHU5LQQZWA, seg_pos={0 0}, upload=441432]: Done uploading piece. err: <nil>
Dec 26 14:30:29.019924030 (503h1m28.891007075s) [noise=true , address=145.79.227.16:4910 , piece_id=IYSTF4W4CG2VNOVSN3QZE2EN6DAMGUEFREC6TYHYBLMQ77ZJSAUQ, seg_pos={0 0}, upload=441432]: Done uploading piece. err: <nil>
Dec 26 14:30:29.027674258 (503h1m28.898757303s) [noise=true , address=145.79.227.21:4475 , piece_id=F7A3TCHUH7GUOBWJBDMLPBU2OJAXP6NAF7T5SCXD4CQS4SDDIBXQ, seg_pos={0 0}, upload=441432]: Done uploading piece. err: <nil>
Dec 26 14:30:29.028119435 (503h1m28.89920248s) [noise=true , address=145.79.227.46:4603 , piece_id=AYJUD7SZI5RL3BVITMSJTXUEEJQY762MMN6Z7W2WZ5VBMWVRCCWA, seg_pos={0 0}, upload=441432]: Done uploading piece. err: <nil>
Dec 26 14:30:29.043452513 (503h1m28.914535558s) [noise=true , address=145.79.227.13:4803 , piece_id=EJEGRDHJ4YSB2556QLODTRDGCM2RFDRNJZTC4NKJ4YLFYBQNLWAQ, seg_pos={0 0}, upload=441432]: Done uploading piece. err: <nil>
Dec 26 14:30:29.043871070 (503h1m28.914954115s) [noise=true , address=145.79.227.47:4266 , piece_id=5YHHH3DYLFYZCIXMJOIH3TJZVLOOPDTVX4Y3P3WPRM4DLBEWFX6A, seg_pos={0 0}, upload=441432]: Done uploading piece. err: <nil>
Dec 26 14:30:29.055245158 (503h1m28.926328193s) [noise=true , address=145.79.227.38:4977 , piece_id=ONNO4SRTWCNKXPLXSQLTYI2VZISLTAYTNCKBEMDIXL3P7FWAJJTQ, seg_pos={0 0}, upload=441432]: Done uploading piece. err: <nil>
Dec 26 14:30:29.288803276 (503h1m29.159886321s) [noise=true , address=45.91.116.10:4082 , piece_id=M5SLIV4MAQ23GMAAXLG3MOSITXOOXCV3UJ32HXYFYQQACHVDRIOQ, seg_pos={0 0}, upload=441432]: Done uploading piece. err: <nil>
Dec 26 14:30:29.336517823 (503h1m29.207600858s) [noise=true , address=45.91.116.5:4298 , piece_id=VW6YPVYORDEMOD5JU2FKD5CJZGFCSTLW5Y235YOVSUP6JX2CBMRQ, seg_pos={0 0}, upload=441432]: Done uploading piece. err: <nil>
Dec 26 14:30:29.358057719 (503h1m29.229140764s) [noise=true , address=45.91.116.13:4155 , piece_id=URJDHIOTOEQDK5MJVYT5LVXL3AXKLGUJVIK6PLQGRBYYGC3LWMJA, seg_pos={0 0}, upload=441432]: Done uploading piece. err: <nil>
Dec 26 14:30:29.443687072 (503h1m29.314770117s) [noise=true , address=45.91.116.7:4009 , piece_id=CZF2WLV3DLMICJMKBLUVCNU4KBB5WSEL5DW2ELLZNODFIYA7XFUQ, seg_pos={0 0}, upload=441432]: Done uploading piece. err: <nil>
Dec 26 14:30:29.443706048 (503h1m29.314789083s) [seg_pos={0 0}, upload=441432]: Segment reached optimal threshold of 13 pieces.
Dec 26 14:30:29.443892994 (503h1m29.314976029s) [noise=true , address=145.79.227.11:4732 , piece_id=G3SYKWBI4P6WET75NQE7CZS6E23RSWVFH2JAYZBXMSW2SBODXAMQ, seg_pos={0 0}, upload=441432]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: expected piece hash; context canceled; EOF
Dec 26 14:30:29.443899357 (503h1m29.314982402s) [noise=true , address=145.79.227.11:4732 , piece_id=G3SYKWBI4P6WET75NQE7CZS6E23RSWVFH2JAYZBXMSW2SBODXAMQ, seg_pos={0 0}, upload=441432]: Long tail cancellation due to optimal threshold
Dec 26 14:30:29.444091773 (503h1m29.315174808s) [noise=true , address=45.91.116.12:4227 , piece_id=OR6YGGDRMUEZLDHU4YWSI2L7UBA5HLDRMIKYKYWWOMAZTUVIJMBQ, seg_pos={0 0}, upload=441432]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: write tcp 188.166.225.170:51518->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51518->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51518->45.91.116.12:4227: use of closed network connection
Dec 26 14:30:29.444100339 (503h1m29.315183384s) [noise=true , address=45.91.116.12:4227 , piece_id=OR6YGGDRMUEZLDHU4YWSI2L7UBA5HLDRMIKYKYWWOMAZTUVIJMBQ, seg_pos={0 0}, upload=441432]: Long tail cancellation due to optimal threshold
Dec 26 14:30:29.444159742 (503h1m29.315242777s) [noise=true , address=45.91.116.14:4190 , piece_id=66J7YGDQ7KUVERKEO5AOXCIAGWZSSJAKE7CNBCNMQ2SADH5OJOUQ, seg_pos={0 0}, upload=441432]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: read tcp 188.166.225.170:34952->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 14:30:29.444178058 (503h1m29.315261103s) [noise=true , address=45.91.116.14:4190 , piece_id=66J7YGDQ7KUVERKEO5AOXCIAGWZSSJAKE7CNBCNMQ2SADH5OJOUQ, seg_pos={0 0}, upload=441432]: Long tail cancellation due to optimal threshold
Dec 26 14:30:29.444218234 (503h1m29.315301289s) [seg_pos={0 0}, upload=441432]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 14:30:29.444327091 (503h1m29.315410126s) [upload=441432]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:30:29.444333293 (503h1m29.315416338s) [upload=441432]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:30:29.729838507 (503h1m29.600921562s) [upload=441432]: Done uploading segments...
Dec 26 14:30:29.729865729 (503h1m29.600948774s) [upload=441432]: Upload finished. err: <nil>
Duration: 7.037994937s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
13.505709888s trace:
https://jaeger.dev.storj.io/trace/3ccdb98964ad100c
FAIL (2025-12-26 14:31:03 UTC)
orbiter warning: context deadline exceeded
Duration: 13.505751717s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
9.698534169s trace:
https://jaeger.dev.storj.io/trace/5b63846e68ea1b18
FAIL (2025-12-26 14:31:03 UTC)
orbiter warning: context deadline exceeded
Duration: 9.698567464s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
54 out of 60 is failed (threshold=6) [X.XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX.XXXXX.XXXXXXX.XXXXXXXX] duration:
5.001517893s trace:
https://jaeger.dev.storj.io/trace/1bc94866eda0715d
FAIL (2025-12-26 14:30:59 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001545226s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [..XX..XX...X...X...........X..X.X...........X....XX.XXX..XXX] duration:
30.003072699s trace:
https://jaeger.dev.storj.io/trace/591402ad96a55046
FAIL (2025-12-26 14:30:56 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Starting upload
Dec 26 14:30:26.764451839 (503h1m26.635534874s) [upload=441432]: Done starting upload
Dec 26 14:30:26.764500713 (503h1m26.635583748s) [upload=441432]: Uploading segments...
Dec 26 14:30:26.764633014 (503h1m26.635716049s) [upload=441432]: Got next segment. Inline: false
Dec 26 14:30:26.764670596 (503h1m26.635753652s) [upload=441432]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 14:30:26.764676618 (503h1m26.635759663s) [upload=441432]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 14:30:27.009506388 (503h1m26.880589443s) [seg_pos={0 0}, upload=441432]: Begin upload segment...
Dec 26 14:30:27.009571281 (503h1m26.880654327s) [seg_pos={0 0}, upload=441432]: Done begin upload segment.
Dec 26 14:30:27.009613822 (503h1m26.880696867s) [noise=true , address=145.79.227.39:4015 , piece_id=B2EZYPCFXBXODQXFEDKJ5UBANRQRCHY52737EYOMXVT45DAYXZ4Q, seg_pos={0 0}, upload=441432]: Uploading piece...
Dec 26 14:30:27.009634261 (503h1m26.880717306s) [noise=true , address=145.79.227.42:4121 , piece_id=U2JUFVVRCFQ6QUXZONLP4E7GW6E6YYBUF4UBEOA2L22M2TM3MLGA, seg_pos={0 0}, upload=441432]: Uploading piece...
Dec 26 14:30:27.010148059 (503h1m26.881231094s) [noise=true , address=145.79.227.11:4732 , piece_id=G3SYKWBI4P6WET75NQE7CZS6E23RSWVFH2JAYZBXMSW2SBODXAMQ, seg_pos={0 0}, upload=441432]: Uploading piece...
Dec 26 14:30:27.010220778 (503h1m26.881303813s) [noise=true , address=145.79.227.38:4977 , piece_id=ONNO4SRTWCNKXPLXSQLTYI2VZISLTAYTNCKBEMDIXL3P7FWAJJTQ, seg_pos={0 0}, upload=441432]: Uploading piece...
Dec 26 14:30:27.010697746 (503h1m26.881780791s) [noise=true , address=45.91.116.5:4298 , piece_id=VW6YPVYORDEMOD5JU2FKD5CJZGFCSTLW5Y235YOVSUP6JX2CBMRQ, seg_pos={0 0}, upload=441432]: Uploading piece...
Dec 26 14:30:27.011130149 (503h1m26.882213184s) [noise=true , address=145.79.227.47:4266 , piece_id=5YHHH3DYLFYZCIXMJOIH3TJZVLOOPDTVX4Y3P3WPRM4DLBEWFX6A, seg_pos={0 0}, upload=441432]: Uploading piece...
Dec 26 14:30:27.012038069 (503h1m26.883121114s) [noise=true , address=145.79.227.13:4803 , piece_id=EJEGRDHJ4YSB2556QLODTRDGCM2RFDRNJZTC4NKJ4YLFYBQNLWAQ, seg_pos={0 0}, upload=441432]: Uploading piece...
Dec 26 14:30:27.012798265 (503h1m26.883881311s) [noise=true , address=45.91.116.13:4155 , piece_id=URJDHIOTOEQDK5MJVYT5LVXL3AXKLGUJVIK6PLQGRBYYGC3LWMJA, seg_pos={0 0}, upload=441432]: Uploading piece...
Dec 26 14:30:27.013487919 (503h1m26.884570954s) [noise=true , address=45.91.116.7:4009 , piece_id=CZF2WLV3DLMICJMKBLUVCNU4KBB5WSEL5DW2ELLZNODFIYA7XFUQ, seg_pos={0 0}, upload=441432]: Uploading piece...
Dec 26 14:30:27.014309634 (503h1m26.885392679s) [noise=true , address=145.79.227.46:4603 , piece_id=AYJUD7SZI5RL3BVITMSJTXUEEJQY762MMN6Z7W2WZ5VBMWVRCCWA, seg_pos={0 0}, upload=441432]: Uploading piece...
Dec 26 14:30:27.014692703 (503h1m26.885775738s) [noise=true , address=145.79.227.6:4257 , piece_id=OWSV6LYJGXYBOH3QVNU22DZVOFJQLZMZA3KKBIKQKXWHU5LQQZWA, seg_pos={0 0}, upload=441432]: Uploading piece...
Dec 26 14:30:27.015114136 (503h1m26.886197181s) [noise=true , address=145.79.227.16:4910 , piece_id=IYSTF4W4CG2VNOVSN3QZE2EN6DAMGUEFREC6TYHYBLMQ77ZJSAUQ, seg_pos={0 0}, upload=441432]: Uploading piece...
Dec 26 14:30:27.015577458 (503h1m26.886660503s) [noise=true , address=45.91.116.10:4082 , piece_id=M5SLIV4MAQ23GMAAXLG3MOSITXOOXCV3UJ32HXYFYQQACHVDRIOQ, seg_pos={0 0}, upload=441432]: Uploading piece...
Dec 26 14:30:27.016058424 (503h1m26.887141459s) [noise=true , address=145.79.227.21:4475 , piece_id=F7A3TCHUH7GUOBWJBDMLPBU2OJAXP6NAF7T5SCXD4CQS4SDDIBXQ, seg_pos={0 0}, upload=441432]: Uploading piece...
Dec 26 14:30:27.016527687 (503h1m26.887610722s) [noise=true , address=45.91.116.12:4227 , piece_id=OR6YGGDRMUEZLDHU4YWSI2L7UBA5HLDRMIKYKYWWOMAZTUVIJMBQ, seg_pos={0 0}, upload=441432]: Uploading piece...
Dec 26 14:30:27.016858367 (503h1m26.887941402s) [noise=true , address=45.91.116.14:4190 , piece_id=66J7YGDQ7KUVERKEO5AOXCIAGWZSSJAKE7CNBCNMQ2SADH5OJOUQ, seg_pos={0 0}, upload=441432]: Uploading piece...
Dec 26 14:30:28.722398527 (503h1m28.593481562s) [upload=441432]: Next returned nil segment
Dec 26 14:30:28.722412724 (503h1m28.593495759s) [upload=441432]: Waiting for error group managing segments...
Dec 26 14:30:29.011610900 (503h1m28.882693945s) [noise=true , address=145.79.227.39:4015 , piece_id=B2EZYPCFXBXODQXFEDKJ5UBANRQRCHY52737EYOMXVT45DAYXZ4Q, seg_pos={0 0}, upload=441432]: Done uploading piece. err: <nil>
Dec 26 14:30:29.017864138 (503h1m28.888947173s) [noise=true , address=145.79.227.42:4121 , piece_id=U2JUFVVRCFQ6QUXZONLP4E7GW6E6YYBUF4UBEOA2L22M2TM3MLGA, seg_pos={0 0}, upload=441432]: Done uploading piece. err: <nil>
Dec 26 14:30:29.019890516 (503h1m28.890973561s) [noise=true , address=145.79.227.6:4257 , piece_id=OWSV6LYJGXYBOH3QVNU22DZVOFJQLZMZA3KKBIKQKXWHU5LQQZWA, seg_pos={0 0}, upload=441432]: Done uploading piece. err: <nil>
Dec 26 14:30:29.019924030 (503h1m28.891007075s) [noise=true , address=145.79.227.16:4910 , piece_id=IYSTF4W4CG2VNOVSN3QZE2EN6DAMGUEFREC6TYHYBLMQ77ZJSAUQ, seg_pos={0 0}, upload=441432]: Done uploading piece. err: <nil>
Dec 26 14:30:29.027674258 (503h1m28.898757303s) [noise=true , address=145.79.227.21:4475 , piece_id=F7A3TCHUH7GUOBWJBDMLPBU2OJAXP6NAF7T5SCXD4CQS4SDDIBXQ, seg_pos={0 0}, upload=441432]: Done uploading piece. err: <nil>
Dec 26 14:30:29.028119435 (503h1m28.89920248s) [noise=true , address=145.79.227.46:4603 , piece_id=AYJUD7SZI5RL3BVITMSJTXUEEJQY762MMN6Z7W2WZ5VBMWVRCCWA, seg_pos={0 0}, upload=441432]: Done uploading piece. err: <nil>
Dec 26 14:30:29.043452513 (503h1m28.914535558s) [noise=true , address=145.79.227.13:4803 , piece_id=EJEGRDHJ4YSB2556QLODTRDGCM2RFDRNJZTC4NKJ4YLFYBQNLWAQ, seg_pos={0 0}, upload=441432]: Done uploading piece. err: <nil>
Dec 26 14:30:29.043871070 (503h1m28.914954115s) [noise=true , address=145.79.227.47:4266 , piece_id=5YHHH3DYLFYZCIXMJOIH3TJZVLOOPDTVX4Y3P3WPRM4DLBEWFX6A, seg_pos={0 0}, upload=441432]: Done uploading piece. err: <nil>
Dec 26 14:30:29.055245158 (503h1m28.926328193s) [noise=true , address=145.79.227.38:4977 , piece_id=ONNO4SRTWCNKXPLXSQLTYI2VZISLTAYTNCKBEMDIXL3P7FWAJJTQ, seg_pos={0 0}, upload=441432]: Done uploading piece. err: <nil>
Dec 26 14:30:29.288803276 (503h1m29.159886321s) [noise=true , address=45.91.116.10:4082 , piece_id=M5SLIV4MAQ23GMAAXLG3MOSITXOOXCV3UJ32HXYFYQQACHVDRIOQ, seg_pos={0 0}, upload=441432]: Done uploading piece. err: <nil>
Dec 26 14:30:29.336517823 (503h1m29.207600858s) [noise=true , address=45.91.116.5:4298 , piece_id=VW6YPVYORDEMOD5JU2FKD5CJZGFCSTLW5Y235YOVSUP6JX2CBMRQ, seg_pos={0 0}, upload=441432]: Done uploading piece. err: <nil>
Dec 26 14:30:29.358057719 (503h1m29.229140764s) [noise=true , address=45.91.116.13:4155 , piece_id=URJDHIOTOEQDK5MJVYT5LVXL3AXKLGUJVIK6PLQGRBYYGC3LWMJA, seg_pos={0 0}, upload=441432]: Done uploading piece. err: <nil>
Dec 26 14:30:29.443687072 (503h1m29.314770117s) [noise=true , address=45.91.116.7:4009 , piece_id=CZF2WLV3DLMICJMKBLUVCNU4KBB5WSEL5DW2ELLZNODFIYA7XFUQ, seg_pos={0 0}, upload=441432]: Done uploading piece. err: <nil>
Dec 26 14:30:29.443706048 (503h1m29.314789083s) [seg_pos={0 0}, upload=441432]: Segment reached optimal threshold of 13 pieces.
Dec 26 14:30:29.443892994 (503h1m29.314976029s) [noise=true , address=145.79.227.11:4732 , piece_id=G3SYKWBI4P6WET75NQE7CZS6E23RSWVFH2JAYZBXMSW2SBODXAMQ, seg_pos={0 0}, upload=441432]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: expected piece hash; context canceled; EOF
Dec 26 14:30:29.443899357 (503h1m29.314982402s) [noise=true , address=145.79.227.11:4732 , piece_id=G3SYKWBI4P6WET75NQE7CZS6E23RSWVFH2JAYZBXMSW2SBODXAMQ, seg_pos={0 0}, upload=441432]: Long tail cancellation due to optimal threshold
Dec 26 14:30:29.444091773 (503h1m29.315174808s) [noise=true , address=45.91.116.12:4227 , piece_id=OR6YGGDRMUEZLDHU4YWSI2L7UBA5HLDRMIKYKYWWOMAZTUVIJMBQ, seg_pos={0 0}, upload=441432]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: write tcp 188.166.225.170:51518->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51518->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51518->45.91.116.12:4227: use of closed network connection
Dec 26 14:30:29.444100339 (503h1m29.315183384s) [noise=true , address=45.91.116.12:4227 , piece_id=OR6YGGDRMUEZLDHU4YWSI2L7UBA5HLDRMIKYKYWWOMAZTUVIJMBQ, seg_pos={0 0}, upload=441432]: Long tail cancellation due to optimal threshold
Dec 26 14:30:29.444159742 (503h1m29.315242777s) [noise=true , address=45.91.116.14:4190 , piece_id=66J7YGDQ7KUVERKEO5AOXCIAGWZSSJAKE7CNBCNMQ2SADH5OJOUQ, seg_pos={0 0}, upload=441432]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: read tcp 188.166.225.170:34952->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 14:30:29.444178058 (503h1m29.315261103s) [noise=true , address=45.91.116.14:4190 , piece_id=66J7YGDQ7KUVERKEO5AOXCIAGWZSSJAKE7CNBCNMQ2SADH5OJOUQ, seg_pos={0 0}, upload=441432]: Long tail cancellation due to optimal threshold
Dec 26 14:30:29.444218234 (503h1m29.315301289s) [seg_pos={0 0}, upload=441432]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 14:30:29.444327091 (503h1m29.315410126s) [upload=441432]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:30:29.444333293 (503h1m29.315416338s) [upload=441432]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:30:29.729838507 (503h1m29.600921562s) [upload=441432]: Done uploading segments...
Dec 26 14:30:29.729865729 (503h1m29.600948774s) [upload=441432]: Upload finished. err: <nil>
Duration: 30.003087037s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
16.174821173s trace:
https://jaeger.dev.storj.io/trace/5b63846e68ea1b18
FAIL (2025-12-26 14:30:51 UTC)
orbiter warning: context deadline exceeded
Duration: 16.174858846s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
54 out of 60 is failed (threshold=6) [XX.XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX.XXXXX.XXXXXXX.XXXXXXX] duration:
5.001595624s trace:
https://jaeger.dev.storj.io/trace/69547ed189d3da2e
FAIL (2025-12-26 14:30:36 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001607947s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [X..XX..XX...X...X...........X..X.X...........X....XX.XXX..XX] duration:
30.003088805s trace:
https://jaeger.dev.storj.io/trace/61edbe3c1af2b96a
FAIL (2025-12-26 14:30:25 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: VMB3FYVNCODACOHW2673BZ7V4F3SIDOTYP6UUND2M6K6PZ3PVKIA) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: SFZS22IK3X2NR23VPZNNJEDJAMV6IYK2FH5TRUKQ22PW3GZC2J3A) context deadline exceeded
======LOGS======
441424]: Starting upload
Dec 26 14:29:56.515924925 (503h0m56.38700797s) [upload=441424]: Done starting upload
Dec 26 14:29:56.516007813 (503h0m56.387090858s) [upload=441424]: Uploading segments...
Dec 26 14:29:56.516144404 (503h0m56.387227439s) [upload=441424]: Got next segment. Inline: false
Dec 26 14:29:56.516189078 (503h0m56.387272113s) [upload=441424]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 14:29:56.516194398 (503h0m56.387277433s) [upload=441424]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 14:29:56.761479356 (503h0m56.632562391s) [seg_pos={0 0}, upload=441424]: Begin upload segment...
Dec 26 14:29:56.761519924 (503h0m56.632602959s) [seg_pos={0 0}, upload=441424]: Done begin upload segment.
Dec 26 14:29:56.761546644 (503h0m56.632629679s) [noise=true , address=145.79.227.3:4150 , piece_id=4TMDK5SJ2MFLP4G4FKYVXUEFJS357U47A27MCOOJT3XIP5HL2N2Q, seg_pos={0 0}, upload=441424]: Uploading piece...
Dec 26 14:29:56.761560151 (503h0m56.632643186s) [noise=true , address=45.91.116.6:4976 , piece_id=VMB3FYVNCODACOHW2673BZ7V4F3SIDOTYP6UUND2M6K6PZ3PVKIA, seg_pos={0 0}, upload=441424]: Uploading piece...
Dec 26 14:29:56.761727830 (503h0m56.632810865s) [noise=true , address=145.79.227.48:4301 , piece_id=COSG4U5G6FOX7WFI7444YYRPIV2LFSSIVC75U4CH3ULKMHC5X32Q, seg_pos={0 0}, upload=441424]: Uploading piece...
Dec 26 14:29:56.762158720 (503h0m56.633241755s) [noise=true , address=77.90.31.14:4661 , piece_id=5NAEUXPBAS27SHCITEOARUZELUX6MQNNVGRPDTGCZ3WIQMLSQHDQ, seg_pos={0 0}, upload=441424]: Uploading piece...
Dec 26 14:29:56.762489369 (503h0m56.633572404s) [noise=true , address=77.90.31.13:4629 , piece_id=MUMSIF34LR3CXO6PNQERDYN4LV3CB6IGFR5LPE3QRJ5ZT6YFLUFA, seg_pos={0 0}, upload=441424]: Uploading piece...
Dec 26 14:29:56.762844436 (503h0m56.633927471s) [noise=true , address=145.79.227.19:4077 , piece_id=L3CQU5G3QJAUENVV7CT3RFEMGOW3ZPSA5SXXUYKLUJ5CHL7UW4GA, seg_pos={0 0}, upload=441424]: Uploading piece...
Dec 26 14:29:56.763222125 (503h0m56.634305161s) [noise=true , address=145.79.227.51:4411 , piece_id=ZHVBLDKFXYFHIIKCKKSFA2CHZ67FMAOTNGM4FC6RT46B5VKB46NQ, seg_pos={0 0}, upload=441424]: Uploading piece...
Dec 26 14:29:56.763326013 (503h0m56.634409048s) [noise=true , address=145.79.227.36:4518 , piece_id=XTJOALVVA3KPJACZP4BR7M3EBMNK2R5QWCCJODPIXZWI7VQRNMGQ, seg_pos={0 0}, upload=441424]: Uploading piece...
Dec 26 14:29:56.763946103 (503h0m56.635029138s) [noise=true , address=145.79.227.21:4473 , piece_id=POZY6WU7OM4U7RLDHB5KGTBW2LJM4UG5LCYN7PAUA2VBWV73TRVQ, seg_pos={0 0}, upload=441424]: Uploading piece...
Dec 26 14:29:56.764356505 (503h0m56.63543954s) [noise=true , address=45.91.116.10:4084 , piece_id=SFZS22IK3X2NR23VPZNNJEDJAMV6IYK2FH5TRUKQ22PW3GZC2J3A, seg_pos={0 0}, upload=441424]: Uploading piece...
Dec 26 14:29:56.764688257 (503h0m56.635771292s) [noise=true , address=145.79.227.13:4803 , piece_id=EZWBEZ5ZZDYHJM6OFX4SBQLXAZFHDBNZNHSQZTNZRNUFBOEYWIHQ, seg_pos={0 0}, upload=441424]: Uploading piece...
Dec 26 14:29:56.765537644 (503h0m56.636620679s) [noise=true , address=145.79.227.47:4265 , piece_id=F3H5ZDX3L2HXMCBYY576YBI4LWS365KSKVJHUZJ54K3MNBTV54EA, seg_pos={0 0}, upload=441424]: Uploading piece...
Dec 26 14:29:56.767085252 (503h0m56.638168287s) [noise=true , address=145.79.227.15:4875 , piece_id=YXU6VHSCZMTILOYTBOSVWQ5BN4XZVMN7KMSQ2VV6FATWPNPILLXQ, seg_pos={0 0}, upload=441424]: Uploading piece...
Dec 26 14:29:56.767565195 (503h0m56.63864824s) [noise=true , address=145.79.227.39:4015 , piece_id=3CCAUHPPKIVBJG2SPYVRZ7KGAOF7BUP62YQKXXYNTKKIRHKIIH5Q, seg_pos={0 0}, upload=441424]: Uploading piece...
Dec 26 14:29:56.768781451 (503h0m56.639864486s) [noise=true , address=145.79.227.41:4087 , piece_id=SXB24LTM2GJQAX5ARD4F5QUYVDJJKFK5S35S4NANRAO55X2QK4TA, seg_pos={0 0}, upload=441424]: Uploading piece...
Dec 26 14:29:56.769930819 (503h0m56.641013854s) [noise=true , address=77.90.31.17:4770 , piece_id=HKKFE7BRDKJVRJ3DZJ75EIFQOP6ZUOIDG7OHF2T2FF3XZPG753DQ, seg_pos={0 0}, upload=441424]: Uploading piece...
Dec 26 14:29:58.496226202 (503h0m58.367309237s) [upload=441424]: Next returned nil segment
Dec 26 14:29:58.496245027 (503h0m58.367328062s) [upload=441424]: Waiting for error group managing segments...
Dec 26 14:29:58.777237588 (503h0m58.648320633s) [noise=true , address=145.79.227.39:4015 , piece_id=3CCAUHPPKIVBJG2SPYVRZ7KGAOF7BUP62YQKXXYNTKKIRHKIIH5Q, seg_pos={0 0}, upload=441424]: Done uploading piece. err: <nil>
Dec 26 14:29:58.780857632 (503h0m58.651940667s) [noise=true , address=145.79.227.3:4150 , piece_id=4TMDK5SJ2MFLP4G4FKYVXUEFJS357U47A27MCOOJT3XIP5HL2N2Q, seg_pos={0 0}, upload=441424]: Done uploading piece. err: <nil>
Dec 26 14:29:58.781998243 (503h0m58.653081288s) [noise=true , address=145.79.227.13:4803 , piece_id=EZWBEZ5ZZDYHJM6OFX4SBQLXAZFHDBNZNHSQZTNZRNUFBOEYWIHQ, seg_pos={0 0}, upload=441424]: Done uploading piece. err: <nil>
Dec 26 14:29:58.790602878 (503h0m58.661685913s) [noise=true , address=145.79.227.19:4077 , piece_id=L3CQU5G3QJAUENVV7CT3RFEMGOW3ZPSA5SXXUYKLUJ5CHL7UW4GA, seg_pos={0 0}, upload=441424]: Done uploading piece. err: <nil>
Dec 26 14:29:58.790748545 (503h0m58.661831581s) [noise=true , address=145.79.227.41:4087 , piece_id=SXB24LTM2GJQAX5ARD4F5QUYVDJJKFK5S35S4NANRAO55X2QK4TA, seg_pos={0 0}, upload=441424]: Done uploading piece. err: <nil>
Dec 26 14:29:58.791581432 (503h0m58.662664477s) [noise=true , address=145.79.227.15:4875 , piece_id=YXU6VHSCZMTILOYTBOSVWQ5BN4XZVMN7KMSQ2VV6FATWPNPILLXQ, seg_pos={0 0}, upload=441424]: Done uploading piece. err: <nil>
Dec 26 14:29:58.792173629 (503h0m58.663256674s) [noise=true , address=145.79.227.48:4301 , piece_id=COSG4U5G6FOX7WFI7444YYRPIV2LFSSIVC75U4CH3ULKMHC5X32Q, seg_pos={0 0}, upload=441424]: Done uploading piece. err: <nil>
Dec 26 14:29:58.796951528 (503h0m58.668034563s) [noise=true , address=145.79.227.21:4473 , piece_id=POZY6WU7OM4U7RLDHB5KGTBW2LJM4UG5LCYN7PAUA2VBWV73TRVQ, seg_pos={0 0}, upload=441424]: Done uploading piece. err: <nil>
Dec 26 14:29:58.798562375 (503h0m58.66964541s) [noise=true , address=145.79.227.51:4411 , piece_id=ZHVBLDKFXYFHIIKCKKSFA2CHZ67FMAOTNGM4FC6RT46B5VKB46NQ, seg_pos={0 0}, upload=441424]: Done uploading piece. err: <nil>
Dec 26 14:29:58.800844049 (503h0m58.671927084s) [noise=true , address=145.79.227.36:4518 , piece_id=XTJOALVVA3KPJACZP4BR7M3EBMNK2R5QWCCJODPIXZWI7VQRNMGQ, seg_pos={0 0}, upload=441424]: Done uploading piece. err: <nil>
Dec 26 14:29:58.813408805 (503h0m58.68449184s) [noise=true , address=145.79.227.47:4265 , piece_id=F3H5ZDX3L2HXMCBYY576YBI4LWS365KSKVJHUZJ54K3MNBTV54EA, seg_pos={0 0}, upload=441424]: Done uploading piece. err: <nil>
Dec 26 14:29:59.023045662 (503h0m58.894128697s) [noise=true , address=45.91.116.6:4976 , piece_id=VMB3FYVNCODACOHW2673BZ7V4F3SIDOTYP6UUND2M6K6PZ3PVKIA, seg_pos={0 0}, upload=441424]: Done uploading piece. err: <nil>
Dec 26 14:29:59.029616945 (503h0m58.90069999s) [noise=true , address=45.91.116.10:4084 , piece_id=SFZS22IK3X2NR23VPZNNJEDJAMV6IYK2FH5TRUKQ22PW3GZC2J3A, seg_pos={0 0}, upload=441424]: Done uploading piece. err: <nil>
Dec 26 14:29:59.029657392 (503h0m58.900740437s) [seg_pos={0 0}, upload=441424]: Segment reached optimal threshold of 13 pieces.
Dec 26 14:29:59.029999003 (503h0m58.901082038s) [noise=true , address=77.90.31.17:4770 , piece_id=HKKFE7BRDKJVRJ3DZJ75EIFQOP6ZUOIDG7OHF2T2FF3XZPG753DQ, seg_pos={0 0}, upload=441424]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF
Dec 26 14:29:59.030006707 (503h0m58.901089742s) [noise=true , address=77.90.31.17:4770 , piece_id=HKKFE7BRDKJVRJ3DZJ75EIFQOP6ZUOIDG7OHF2T2FF3XZPG753DQ, seg_pos={0 0}, upload=441424]: Long tail cancellation due to optimal threshold
Dec 26 14:29:59.030153466 (503h0m58.901236512s) [noise=true , address=77.90.31.14:4661 , piece_id=5NAEUXPBAS27SHCITEOARUZELUX6MQNNVGRPDTGCZ3WIQMLSQHDQ, seg_pos={0 0}, upload=441424]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:37384->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:37384->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37384->77.90.31.14:4661: use of closed network connection
Dec 26 14:29:59.030188052 (503h0m58.901271087s) [noise=true , address=77.90.31.14:4661 , piece_id=5NAEUXPBAS27SHCITEOARUZELUX6MQNNVGRPDTGCZ3WIQMLSQHDQ, seg_pos={0 0}, upload=441424]: Long tail cancellation due to optimal threshold
Dec 26 14:29:59.030309283 (503h0m58.901392338s) [noise=true , address=77.90.31.13:4629 , piece_id=MUMSIF34LR3CXO6PNQERDYN4LV3CB6IGFR5LPE3QRJ5ZT6YFLUFA, seg_pos={0 0}, upload=441424]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: read tcp 188.166.225.170:35614->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 14:29:59.030344460 (503h0m58.901427495s) [noise=true , address=77.90.31.13:4629 , piece_id=MUMSIF34LR3CXO6PNQERDYN4LV3CB6IGFR5LPE3QRJ5ZT6YFLUFA, seg_pos={0 0}, upload=441424]: Long tail cancellation due to optimal threshold
Dec 26 14:29:59.030393163 (503h0m58.901476208s) [seg_pos={0 0}, upload=441424]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 14:29:59.030481541 (503h0m58.901564586s) [upload=441424]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:29:59.030487182 (503h0m58.901570217s) [upload=441424]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:29:59.323101603 (503h0m59.194184648s) [upload=441424]: Done uploading segments...
Dec 26 14:29:59.323120579 (503h0m59.194203624s) [upload=441424]: Upload finished. err: <nil>
Duration: 30.003099526s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
15.684321694s trace:
https://jaeger.dev.storj.io/trace/5b63846e68ea1b18
FAIL (2025-12-26 14:30:22 UTC)
orbiter warning: context deadline exceeded
Duration: 15.684350359s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
19 out of 60 is failed (threshold=3) [X.X.XX.....X..........X..X..X.X.X....X.....XXX.XX.XX.....X..] duration:
56.69006202s trace:
https://jaeger.dev.storj.io/trace/3b3dc37aa1526738
FAIL (2025-12-26 14:30:12 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1nstPr2GMJo6pnBmBuDME5QYw3mXgnqLoaTKkcKUhjSbAY8mQk, Piece ID: UAASM7Q6N6SJFL3BZQG4J3PWGICDNX7HWKKMLAE3Z4DTLF2CQAHQ) context deadline exceeded; piecestore: (Node ID: 12HcwSEGPEpd59dUukENRoUykRyqmVLiGcKEBbJ72qr9b5tpoEY, Piece ID: WSRZPIBVTNRPGFTBESN6ZEWLITLZOZLBB4IPRAUPYH3M6YEQOWZQ) context deadline exceeded; piecestore: (Node ID: 12QnZ3q9oe4zPuLpS3V1DCBhyaU1umS5B7bpuAqPkaaY96ErWbo, Piece ID: XZSYVBXPVPA73GHQBP7PFICVBSO3RNC3YLIOSYGIPBKQ66FHQPWA) context deadline exceeded; piecestore: (Node ID: 121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, Piece ID: 7GZ6PRPYTSWSCFWRSFIBJ5EKN7FETMBWUYJKSS2YVKG3JJK42S3A) context deadline exceeded; piecestore: (Node ID: 1XjJpLBEZm5daHcBUmUyC7QpLVhu9Vm9ex9RYcYJWEuLGk527F, Piece ID: J2UOIRDKCGLOS6FPEQCM5SZWGPDPRPO6O2HFY2KJMSWTFZEPRAMQ) context deadline exceeded
======LOGS======
7s) [noise=true , address=45.91.116.12:4236 , piece_id=M5R7IAUXCFF7QF4HHWME6H7PKQJYRKBGLGU5VFJJFQE6LUPIYCJA, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:12.729380403 (502h58m12.600463448s) [noise=true , address=77.90.31.21:4909 , piece_id=7GZ6PRPYTSWSCFWRSFIBJ5EKN7FETMBWUYJKSS2YVKG3JJK42S3A, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:12.729753183 (502h58m12.600836228s) [noise=true , address=64.49.13.139:4480 , piece_id=KGAH4B57THDLG3FIMSEKYO2PE5JXKYLBLRMQQH37CKR5YCBL3QGQ, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:12.730155719 (502h58m12.601238764s) [noise=true , address=45.91.116.10:4097 , piece_id=WSRZPIBVTNRPGFTBESN6ZEWLITLZOZLBB4IPRAUPYH3M6YEQOWZQ, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:12.730540132 (502h58m12.601623217s) [noise=true , address=145.79.227.20:4452 , piece_id=UGIUMXQNQIXGGNF2BOQTADA5JOUQKEI27DTWXFB6DIJUJ2GTGBQQ, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:12.730922941 (502h58m12.602005986s) [noise=true , address=145.79.227.40:4076 , piece_id=XVEBWXMTJ7VK2KCPYXS2LDXP2BRI2JYMV6EZF6XMMVTMQ4ICA2BQ, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:12.731266324 (502h58m12.602349369s) [noise=true , address=145.79.227.34:4934 , piece_id=ATCG3GH3MMQSVE5AN6H4JD7WKXPFAOXIR2YIHTHXY2AC2OKPWBMQ, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:12.731693027 (502h58m12.602776062s) [noise=true , address=77.90.31.17:4799 , piece_id=NG3PPYMUUODKZVAVYKGTWZ2QBHOHINP45ZAMOBTQBZBJ6VOPPFNA, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:12.732138174 (502h58m12.603221209s) [noise=true , address=77.90.31.13:4640 , piece_id=J2UOIRDKCGLOS6FPEQCM5SZWGPDPRPO6O2HFY2KJMSWTFZEPRAMQ, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:12.732479955 (502h58m12.603563s) [noise=true , address=64.49.13.27:4957 , piece_id=SI7ANTUA53SPUXSLY2ECQUDHIHWHPHMFTPFWNBMANKZGUW4U2L4A, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:12.732773655 (502h58m12.60385669s) [noise=true , address=77.90.31.18:4832 , piece_id=XZSYVBXPVPA73GHQBP7PFICVBSO3RNC3YLIOSYGIPBKQ66FHQPWA, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:12.733031345 (502h58m12.60411439s) [noise=true , address=64.49.13.169:4711 , piece_id=Q2LQKJ2RBKW2QLHGWIVPWJKBCUQZBQFFOPJGEWMPZZWAILDUJBUQ, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:12.733433241 (502h58m12.604516286s) [noise=true , address=77.90.31.12:4614 , piece_id=5CPNE3UG2AWMOSC2MA6VOSJQLUUDENYYBUQGFHXYN3ON63PVM7EQ, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:12.733723934 (502h58m12.604806969s) [noise=true , address=77.90.31.16:4753 , piece_id=2GMK4UAMU6AHDTQEW447NXYGDKKSZOUFJ7PGL5OGJ5W2GIECTQMQ, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:12.744912929 (502h58m12.615995964s) [noise=true , address=145.79.227.33:4884 , piece_id=JDO3F6HQRBGHWJ74ZVJNML2X5757PP55DWQLF4YT3P7JDKZ7VZIA, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:12.748930370 (502h58m12.620013415s) [noise=true , address=145.79.227.30:4772 , piece_id=MBZYVHFEAHN2FEVS6LHPUCCBHX74E35JO6EOG4GPKBRG7UIK2IPA, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:14.090029017 (502h58m13.961112062s) [upload=441384]: Next returned nil segment
Dec 26 14:27:14.090042262 (502h58m13.961125297s) [upload=441384]: Waiting for error group managing segments...
Dec 26 14:27:14.444533658 (502h58m14.315616693s) [noise=true , address=64.49.13.139:4480 , piece_id=KGAH4B57THDLG3FIMSEKYO2PE5JXKYLBLRMQQH37CKR5YCBL3QGQ, seg_pos={0 0}, upload=441384]: Done uploading piece. err: <nil>
Dec 26 14:27:14.586227048 (502h58m14.457310083s) [noise=true , address=64.49.13.27:4957 , piece_id=SI7ANTUA53SPUXSLY2ECQUDHIHWHPHMFTPFWNBMANKZGUW4U2L4A, seg_pos={0 0}, upload=441384]: Done uploading piece. err: <nil>
Dec 26 14:27:14.622921881 (502h58m14.494004916s) [noise=true , address=145.79.227.20:4452 , piece_id=UGIUMXQNQIXGGNF2BOQTADA5JOUQKEI27DTWXFB6DIJUJ2GTGBQQ, seg_pos={0 0}, upload=441384]: Done uploading piece. err: <nil>
Dec 26 14:27:14.812511214 (502h58m14.683594259s) [noise=true , address=77.90.31.21:4909 , piece_id=7GZ6PRPYTSWSCFWRSFIBJ5EKN7FETMBWUYJKSS2YVKG3JJK42S3A, seg_pos={0 0}, upload=441384]: Done uploading piece. err: <nil>
Dec 26 14:27:14.823469320 (502h58m14.694552355s) [noise=true , address=77.90.31.18:4832 , piece_id=XZSYVBXPVPA73GHQBP7PFICVBSO3RNC3YLIOSYGIPBKQ66FHQPWA, seg_pos={0 0}, upload=441384]: Done uploading piece. err: <nil>
Dec 26 14:27:14.884791981 (502h58m14.755875016s) [noise=true , address=77.90.31.15:4725 , piece_id=UAASM7Q6N6SJFL3BZQG4J3PWGICDNX7HWKKMLAE3Z4DTLF2CQAHQ, seg_pos={0 0}, upload=441384]: Done uploading piece. err: <nil>
Dec 26 14:27:14.897429475 (502h58m14.76851251s) [noise=true , address=77.90.31.12:4614 , piece_id=5CPNE3UG2AWMOSC2MA6VOSJQLUUDENYYBUQGFHXYN3ON63PVM7EQ, seg_pos={0 0}, upload=441384]: Done uploading piece. err: <nil>
Dec 26 14:27:14.921617022 (502h58m14.792700057s) [noise=true , address=77.90.31.13:4640 , piece_id=J2UOIRDKCGLOS6FPEQCM5SZWGPDPRPO6O2HFY2KJMSWTFZEPRAMQ, seg_pos={0 0}, upload=441384]: Done uploading piece. err: <nil>
Dec 26 14:27:15.217935085 (502h58m15.089018121s) [noise=true , address=77.90.31.17:4799 , piece_id=NG3PPYMUUODKZVAVYKGTWZ2QBHOHINP45ZAMOBTQBZBJ6VOPPFNA, seg_pos={0 0}, upload=441384]: Done uploading piece. err: <nil>
Dec 26 14:27:15.538396774 (502h58m15.409479819s) [noise=true , address=45.91.116.10:4097 , piece_id=WSRZPIBVTNRPGFTBESN6ZEWLITLZOZLBB4IPRAUPYH3M6YEQOWZQ, seg_pos={0 0}, upload=441384]: Done uploading piece. err: <nil>
Dec 26 14:27:15.575796018 (502h58m15.446879063s) [noise=true , address=145.79.227.34:4934 , piece_id=ATCG3GH3MMQSVE5AN6H4JD7WKXPFAOXIR2YIHTHXY2AC2OKPWBMQ, seg_pos={0 0}, upload=441384]: Done uploading piece. err: <nil>
Dec 26 14:27:15.588172846 (502h58m15.459255881s) [noise=true , address=145.79.227.30:4772 , piece_id=MBZYVHFEAHN2FEVS6LHPUCCBHX74E35JO6EOG4GPKBRG7UIK2IPA, seg_pos={0 0}, upload=441384]: Done uploading piece. err: <nil>
Dec 26 14:27:15.654034369 (502h58m15.525117404s) [noise=true , address=145.79.227.44:4221 , piece_id=KNE5IJVSCFLIOLLXHC32A4X4DMVQKV6WHWKA7NSWSYYC4F7OMOFA, seg_pos={0 0}, upload=441384]: Done uploading piece. err: <nil>
Dec 26 14:27:15.764565211 (502h58m15.635648246s) [noise=true , address=145.79.227.40:4076 , piece_id=XVEBWXMTJ7VK2KCPYXS2LDXP2BRI2JYMV6EZF6XMMVTMQ4ICA2BQ, seg_pos={0 0}, upload=441384]: Done uploading piece. err: <nil>
Dec 26 14:27:15.764584317 (502h58m15.635667352s) [seg_pos={0 0}, upload=441384]: Segment reached optimal threshold of 14 pieces.
Dec 26 14:27:15.764749782 (502h58m15.635832817s) [noise=true , address=64.49.13.169:4711 , piece_id=Q2LQKJ2RBKW2QLHGWIVPWJKBCUQZBQFFOPJGEWMPZZWAILDUJBUQ, seg_pos={0 0}, upload=441384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:36534->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:36534->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36534->64.49.13.169:4711: use of closed network connection
Dec 26 14:27:15.764755412 (502h58m15.635838447s) [noise=true , address=64.49.13.169:4711 , piece_id=Q2LQKJ2RBKW2QLHGWIVPWJKBCUQZBQFFOPJGEWMPZZWAILDUJBUQ, seg_pos={0 0}, upload=441384]: Long tail cancellation due to optimal threshold
Dec 26 14:27:15.764867776 (502h58m15.635950811s) [noise=true , address=45.91.116.12:4236 , piece_id=M5R7IAUXCFF7QF4HHWME6H7PKQJYRKBGLGU5VFJJFQE6LUPIYCJA, seg_pos={0 0}, upload=441384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH): protocol: expected piece hash; context canceled; EOF
Dec 26 14:27:15.764897062 (502h58m15.635980097s) [noise=true , address=145.79.227.33:4884 , piece_id=JDO3F6HQRBGHWJ74ZVJNML2X5757PP55DWQLF4YT3P7JDKZ7VZIA, seg_pos={0 0}, upload=441384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qGXdnm83P4eMnCtZdRt2x5yAWP7pQafWwoPG72daZJVR9pyKG): protocol: expected piece hash; context canceled; EOF
Dec 26 14:27:15.764903855 (502h58m15.63598689s) [noise=true , address=145.79.227.33:4884 , piece_id=JDO3F6HQRBGHWJ74ZVJNML2X5757PP55DWQLF4YT3P7JDKZ7VZIA, seg_pos={0 0}, upload=441384]: Long tail cancellation due to optimal threshold
Dec 26 14:27:15.764910517 (502h58m15.635993552s) [noise=true , address=45.91.116.12:4236 , piece_id=M5R7IAUXCFF7QF4HHWME6H7PKQJYRKBGLGU5VFJJFQE6LUPIYCJA, seg_pos={0 0}, upload=441384]: Long tail cancellation due to optimal threshold
Dec 26 14:27:15.765067747 (502h58m15.636150782s) [noise=true , address=77.90.31.16:4753 , piece_id=2GMK4UAMU6AHDTQEW447NXYGDKKSZOUFJ7PGL5OGJ5W2GIECTQMQ, seg_pos={0 0}, upload=441384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE): protocol: expected piece hash; context canceled; EOF
Dec 26 14:27:15.765076293 (502h58m15.636159328s) [noise=true , address=77.90.31.16:4753 , piece_id=2GMK4UAMU6AHDTQEW447NXYGDKKSZOUFJ7PGL5OGJ5W2GIECTQMQ, seg_pos={0 0}, upload=441384]: Long tail cancellation due to optimal threshold
Dec 26 14:27:15.765245906 (502h58m15.636328941s) [seg_pos={0 0}, upload=441384]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 14:27:15.765317863 (502h58m15.636400908s) [upload=441384]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:27:15.765322372 (502h58m15.636405407s) [upload=441384]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:27:16.068012309 (502h58m15.939095344s) [upload=441384]: Done uploading segments...
Dec 26 14:27:16.068037878 (502h58m15.939120913s) [upload=441384]: Upload finished. err: <nil>
Duration: 56.69008348s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
14.740554418s trace:
https://jaeger.dev.storj.io/trace/3ccdb98964ad100c
FAIL (2025-12-26 14:30:07 UTC)
orbiter warning: context deadline exceeded
Duration: 14.740582441s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [.X..XX..XX...X...X...........X..X.X...........X....XX.XXX..X] duration:
30.004017573s trace:
https://jaeger.dev.storj.io/trace/50a686330fe81ce6
FAIL (2025-12-26 14:29:55 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: QQS5XWJW42EVLAG7ODODB75VUZDTOJHQUZCVGXMBSGKNC5QJNOFA) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: S3IKIKGJIFROWYTFQG7OKXSGKUHDQFUJ6RPWAM7B7XIIJ2WVPEZA) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: ZTMU5A4ZHI62NGO5EWS24GNJ3T6ZVDHYAK75ZWEGB4E5NQ2MD7SA) context deadline exceeded
======LOGS======
Dec 26 14:29:26.283526224 (503h0m26.154609269s) [upload=441418]: Starting upload
Dec 26 14:29:26.283590847 (503h0m26.154673882s) [upload=441418]: Done starting upload
Dec 26 14:29:26.283850091 (503h0m26.154933126s) [upload=441418]: Uploading segments...
Dec 26 14:29:26.283901298 (503h0m26.154984333s) [upload=441418]: Got next segment. Inline: false
Dec 26 14:29:26.284062605 (503h0m26.15514565s) [upload=441418]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 14:29:26.284069478 (503h0m26.155152523s) [upload=441418]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 14:29:26.535515057 (503h0m26.406598112s) [seg_pos={0 0}, upload=441418]: Begin upload segment...
Dec 26 14:29:26.535569140 (503h0m26.406652185s) [seg_pos={0 0}, upload=441418]: Done begin upload segment.
Dec 26 14:29:26.535612553 (503h0m26.406695598s) [noise=true , address=145.79.227.48:4301 , piece_id=JNUZ7BMCOHITRNYWB5AHBNFD43VQZBSKP2YERFOS6WCKUYZ54H3A, seg_pos={0 0}, upload=441418]: Uploading piece...
Dec 26 14:29:26.535623975 (503h0m26.40670702s) [noise=true , address=145.79.227.37:4567 , piece_id=UENJ7FH2ZPLKPEPSXHQDFNQEKJ6AD2HPEXTM4LUFWXOD2PPT56YQ, seg_pos={0 0}, upload=441418]: Uploading piece...
Dec 26 14:29:26.535739234 (503h0m26.406822269s) [noise=true , address=77.90.31.16:4736 , piece_id=S3IKIKGJIFROWYTFQG7OKXSGKUHDQFUJ6RPWAM7B7XIIJ2WVPEZA, seg_pos={0 0}, upload=441418]: Uploading piece...
Dec 26 14:29:26.536142421 (503h0m26.407225466s) [noise=true , address=145.79.227.5:4222 , piece_id=4KM74YZDSGKWNAPJGRZT2KP374WKFBIGOY6SQHCNNTPPZDADAZ7A, seg_pos={0 0}, upload=441418]: Uploading piece...
Dec 26 14:29:26.536543806 (503h0m26.407626851s) [noise=true , address=145.79.227.42:4122 , piece_id=3IQBYATAI7YSUL2EG6MDGAY2NCHO4NDX4RT6BMDVKQT2M4DPFFIQ, seg_pos={0 0}, upload=441418]: Uploading piece...
Dec 26 14:29:26.536953166 (503h0m26.408036211s) [noise=true , address=45.91.116.14:4189 , piece_id=QKGIEJVPJYP2CD6F2DYGWKJETCKK2G27URJ7OKAFEXK6UTO5AIEA, seg_pos={0 0}, upload=441418]: Uploading piece...
Dec 26 14:29:26.537420966 (503h0m26.408504011s) [noise=true , address=145.79.227.44:4194 , piece_id=CI65XIIAEOG4A743L3YFM7WWXR3W6MIIU53SJQTB76XEUSAS5IGQ, seg_pos={0 0}, upload=441418]: Uploading piece...
Dec 26 14:29:26.537439402 (503h0m26.408522447s) [noise=true , address=77.90.31.21:4880 , piece_id=UF6ZGV3VN4IR6DB57O3LYO7QZAREIEQ463YWEUPY34OABTD6LIHA, seg_pos={0 0}, upload=441418]: Uploading piece...
Dec 26 14:29:26.537762286 (503h0m26.408845331s) [noise=true , address=45.91.116.13:4156 , piece_id=SUCSETJ2ZGGEKQZ4QMNGVY2IKIT7Q323NK5QB45C6IBITZU6R24Q, seg_pos={0 0}, upload=441418]: Uploading piece...
Dec 26 14:29:26.537908154 (503h0m26.408991199s) [noise=true , address=145.79.227.39:4014 , piece_id=BOTA2QKNL3LJO4ARACKB7YNQDRTXDSC4STVHZOW5JLSD5PWHTHKA, seg_pos={0 0}, upload=441418]: Uploading piece...
Dec 26 14:29:26.538180863 (503h0m26.409263908s) [noise=true , address=45.91.116.4:4937 , piece_id=D52JC2FNMJPUBFKELHOWABDRCXFZCQLWDABTXPGLRAIT3ZIMJWBQ, seg_pos={0 0}, upload=441418]: Uploading piece...
Dec 26 14:29:26.538580975 (503h0m26.40966401s) [noise=true , address=145.79.227.45:4229 , piece_id=YCVBIADDCDTWGHKXPGRVILXPSGXAUAG3PYAJEUTTOJWINYJJRDNQ, seg_pos={0 0}, upload=441418]: Uploading piece...
Dec 26 14:29:26.538891085 (503h0m26.40997412s) [noise=true , address=77.90.31.13:4629 , piece_id=QQS5XWJW42EVLAG7ODODB75VUZDTOJHQUZCVGXMBSGKNC5QJNOFA, seg_pos={0 0}, upload=441418]: Uploading piece...
Dec 26 14:29:26.539304422 (503h0m26.410387458s) [noise=true , address=145.79.227.36:4517 , piece_id=UZJNWQSAKUBYEIEH3BLG36VUA5YIJOGBCAZPIJSMLL4FQHOLB7CQ, seg_pos={0 0}, upload=441418]: Uploading piece...
Dec 26 14:29:26.539763026 (503h0m26.410846071s) [noise=true , address=145.79.227.43:4157 , piece_id=FJ66ZYZXTA4GHANNRQLOLLCD3QINRBFK3ZXQHAMCBJZFT42WQMWQ, seg_pos={0 0}, upload=441418]: Uploading piece...
Dec 26 14:29:26.542741848 (503h0m26.413824903s) [noise=true , address=45.91.116.7:4010 , piece_id=ZTMU5A4ZHI62NGO5EWS24GNJ3T6ZVDHYAK75ZWEGB4E5NQ2MD7SA, seg_pos={0 0}, upload=441418]: Uploading piece...
Dec 26 14:29:28.243089019 (503h0m28.114172064s) [upload=441418]: Next returned nil segment
Dec 26 14:29:28.243104498 (503h0m28.114187533s) [upload=441418]: Waiting for error group managing segments...
Dec 26 14:29:28.540139878 (503h0m28.411222933s) [noise=true , address=145.79.227.45:4229 , piece_id=YCVBIADDCDTWGHKXPGRVILXPSGXAUAG3PYAJEUTTOJWINYJJRDNQ, seg_pos={0 0}, upload=441418]: Done uploading piece. err: <nil>
Dec 26 14:29:28.542759306 (503h0m28.413842341s) [noise=true , address=145.79.227.36:4517 , piece_id=UZJNWQSAKUBYEIEH3BLG36VUA5YIJOGBCAZPIJSMLL4FQHOLB7CQ, seg_pos={0 0}, upload=441418]: Done uploading piece. err: <nil>
Dec 26 14:29:28.542787520 (503h0m28.413870565s) [noise=true , address=145.79.227.39:4014 , piece_id=BOTA2QKNL3LJO4ARACKB7YNQDRTXDSC4STVHZOW5JLSD5PWHTHKA, seg_pos={0 0}, upload=441418]: Done uploading piece. err: <nil>
Dec 26 14:29:28.545328518 (503h0m28.416411563s) [noise=true , address=145.79.227.42:4122 , piece_id=3IQBYATAI7YSUL2EG6MDGAY2NCHO4NDX4RT6BMDVKQT2M4DPFFIQ, seg_pos={0 0}, upload=441418]: Done uploading piece. err: <nil>
Dec 26 14:29:28.553525697 (503h0m28.424608742s) [noise=true , address=145.79.227.48:4301 , piece_id=JNUZ7BMCOHITRNYWB5AHBNFD43VQZBSKP2YERFOS6WCKUYZ54H3A, seg_pos={0 0}, upload=441418]: Done uploading piece. err: <nil>
Dec 26 14:29:28.563087385 (503h0m28.43417043s) [noise=true , address=145.79.227.5:4222 , piece_id=4KM74YZDSGKWNAPJGRZT2KP374WKFBIGOY6SQHCNNTPPZDADAZ7A, seg_pos={0 0}, upload=441418]: Done uploading piece. err: <nil>
Dec 26 14:29:28.566169563 (503h0m28.437252608s) [noise=true , address=145.79.227.37:4567 , piece_id=UENJ7FH2ZPLKPEPSXHQDFNQEKJ6AD2HPEXTM4LUFWXOD2PPT56YQ, seg_pos={0 0}, upload=441418]: Done uploading piece. err: <nil>
Dec 26 14:29:28.786314970 (503h0m28.657398005s) [noise=true , address=45.91.116.7:4010 , piece_id=ZTMU5A4ZHI62NGO5EWS24GNJ3T6ZVDHYAK75ZWEGB4E5NQ2MD7SA, seg_pos={0 0}, upload=441418]: Done uploading piece. err: <nil>
Dec 26 14:29:28.794496740 (503h0m28.665579785s) [noise=true , address=45.91.116.13:4156 , piece_id=SUCSETJ2ZGGEKQZ4QMNGVY2IKIT7Q323NK5QB45C6IBITZU6R24Q, seg_pos={0 0}, upload=441418]: Done uploading piece. err: <nil>
Dec 26 14:29:28.825034358 (503h0m28.696117423s) [noise=true , address=77.90.31.16:4736 , piece_id=S3IKIKGJIFROWYTFQG7OKXSGKUHDQFUJ6RPWAM7B7XIIJ2WVPEZA, seg_pos={0 0}, upload=441418]: Done uploading piece. err: <nil>
Dec 26 14:29:29.354761766 (503h0m29.225844801s) [noise=true , address=77.90.31.21:4880 , piece_id=UF6ZGV3VN4IR6DB57O3LYO7QZAREIEQ463YWEUPY34OABTD6LIHA, seg_pos={0 0}, upload=441418]: Done uploading piece. err: <nil>
Dec 26 14:29:29.609604063 (503h0m29.480687098s) [noise=true , address=77.90.31.13:4629 , piece_id=QQS5XWJW42EVLAG7ODODB75VUZDTOJHQUZCVGXMBSGKNC5QJNOFA, seg_pos={0 0}, upload=441418]: Done uploading piece. err: <nil>
Dec 26 14:29:30.135090525 (503h0m30.00617356s) [noise=true , address=45.91.116.4:4937 , piece_id=D52JC2FNMJPUBFKELHOWABDRCXFZCQLWDABTXPGLRAIT3ZIMJWBQ, seg_pos={0 0}, upload=441418]: Done uploading piece. err: <nil>
Dec 26 14:29:30.135109110 (503h0m30.006192146s) [seg_pos={0 0}, upload=441418]: Segment reached optimal threshold of 13 pieces.
Dec 26 14:29:30.135320383 (503h0m30.006403418s) [noise=true , address=45.91.116.14:4189 , piece_id=QKGIEJVPJYP2CD6F2DYGWKJETCKK2G27URJ7OKAFEXK6UTO5AIEA, seg_pos={0 0}, upload=441418]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF
Dec 26 14:29:30.135331013 (503h0m30.006414048s) [noise=true , address=45.91.116.14:4189 , piece_id=QKGIEJVPJYP2CD6F2DYGWKJETCKK2G27URJ7OKAFEXK6UTO5AIEA, seg_pos={0 0}, upload=441418]: Long tail cancellation due to optimal threshold
Dec 26 14:29:30.135492350 (503h0m30.006575385s) [noise=true , address=145.79.227.43:4157 , piece_id=FJ66ZYZXTA4GHANNRQLOLLCD3QINRBFK3ZXQHAMCBJZFT42WQMWQ, seg_pos={0 0}, upload=441418]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF
Dec 26 14:29:30.135508500 (503h0m30.006591536s) [noise=true , address=145.79.227.43:4157 , piece_id=FJ66ZYZXTA4GHANNRQLOLLCD3QINRBFK3ZXQHAMCBJZFT42WQMWQ, seg_pos={0 0}, upload=441418]: Long tail cancellation due to optimal threshold
Dec 26 14:29:30.135537957 (503h0m30.006620992s) [noise=true , address=145.79.227.44:4194 , piece_id=CI65XIIAEOG4A743L3YFM7WWXR3W6MIIU53SJQTB76XEUSAS5IGQ, seg_pos={0 0}, upload=441418]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF
Dec 26 14:29:30.135544128 (503h0m30.006627163s) [noise=true , address=145.79.227.44:4194 , piece_id=CI65XIIAEOG4A743L3YFM7WWXR3W6MIIU53SJQTB76XEUSAS5IGQ, seg_pos={0 0}, upload=441418]: Long tail cancellation due to optimal threshold
Dec 26 14:29:30.135586770 (503h0m30.006669805s) [seg_pos={0 0}, upload=441418]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 14:29:30.135654089 (503h0m30.006737124s) [upload=441418]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:29:30.135658697 (503h0m30.006741742s) [upload=441418]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:29:30.428280262 (503h0m30.299363297s) [upload=441418]: Done uploading segments...
Dec 26 14:29:30.428305239 (503h0m30.299388274s) [upload=441418]: Upload finished. err: <nil>
Duration: 30.00404273s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
10.809455351s trace:
https://jaeger.dev.storj.io/trace/5b63846e68ea1b18
FAIL (2025-12-26 14:29:39 UTC)
orbiter warning: context deadline exceeded
Duration: 10.809474688s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
53 out of 60 is failed (threshold=6) [.XX.XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX.XXXXX.XXXXXXX.XXXXXX] duration:
5.001480112s trace:
https://jaeger.dev.storj.io/trace/312c49d041671bb2
FAIL (2025-12-26 14:29:31 UTC)
orbiter warning: context deadline exceeded
Duration: 5.00150103s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [X.........................................................X.] duration:
15.177959691s trace:
https://jaeger.dev.storj.io/trace/3ccdb98964ad100c
FAIL (2025-12-26 14:29:19 UTC)
orbiter warning: context deadline exceeded
Duration: 15.178024125s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
19 out of 60 is failed (threshold=3) [.X.X.XX.....X..........X..X..X.X.X....X.....XXX.XX.XX.....X.] duration:
56.874523308s trace:
https://jaeger.dev.storj.io/trace/3b3dc37aa1526738
FAIL (2025-12-26 14:29:15 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1nstPr2GMJo6pnBmBuDME5QYw3mXgnqLoaTKkcKUhjSbAY8mQk, Piece ID: UAASM7Q6N6SJFL3BZQG4J3PWGICDNX7HWKKMLAE3Z4DTLF2CQAHQ) context deadline exceeded; piecestore: (Node ID: 12HcwSEGPEpd59dUukENRoUykRyqmVLiGcKEBbJ72qr9b5tpoEY, Piece ID: WSRZPIBVTNRPGFTBESN6ZEWLITLZOZLBB4IPRAUPYH3M6YEQOWZQ) context deadline exceeded; piecestore: (Node ID: 12QnZ3q9oe4zPuLpS3V1DCBhyaU1umS5B7bpuAqPkaaY96ErWbo, Piece ID: XZSYVBXPVPA73GHQBP7PFICVBSO3RNC3YLIOSYGIPBKQ66FHQPWA) context deadline exceeded; piecestore: (Node ID: 121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, Piece ID: 7GZ6PRPYTSWSCFWRSFIBJ5EKN7FETMBWUYJKSS2YVKG3JJK42S3A) context deadline exceeded; piecestore: (Node ID: 1XjJpLBEZm5daHcBUmUyC7QpLVhu9Vm9ex9RYcYJWEuLGk527F, Piece ID: J2UOIRDKCGLOS6FPEQCM5SZWGPDPRPO6O2HFY2KJMSWTFZEPRAMQ) context deadline exceeded
======LOGS======
7s) [noise=true , address=45.91.116.12:4236 , piece_id=M5R7IAUXCFF7QF4HHWME6H7PKQJYRKBGLGU5VFJJFQE6LUPIYCJA, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:12.729380403 (502h58m12.600463448s) [noise=true , address=77.90.31.21:4909 , piece_id=7GZ6PRPYTSWSCFWRSFIBJ5EKN7FETMBWUYJKSS2YVKG3JJK42S3A, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:12.729753183 (502h58m12.600836228s) [noise=true , address=64.49.13.139:4480 , piece_id=KGAH4B57THDLG3FIMSEKYO2PE5JXKYLBLRMQQH37CKR5YCBL3QGQ, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:12.730155719 (502h58m12.601238764s) [noise=true , address=45.91.116.10:4097 , piece_id=WSRZPIBVTNRPGFTBESN6ZEWLITLZOZLBB4IPRAUPYH3M6YEQOWZQ, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:12.730540132 (502h58m12.601623217s) [noise=true , address=145.79.227.20:4452 , piece_id=UGIUMXQNQIXGGNF2BOQTADA5JOUQKEI27DTWXFB6DIJUJ2GTGBQQ, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:12.730922941 (502h58m12.602005986s) [noise=true , address=145.79.227.40:4076 , piece_id=XVEBWXMTJ7VK2KCPYXS2LDXP2BRI2JYMV6EZF6XMMVTMQ4ICA2BQ, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:12.731266324 (502h58m12.602349369s) [noise=true , address=145.79.227.34:4934 , piece_id=ATCG3GH3MMQSVE5AN6H4JD7WKXPFAOXIR2YIHTHXY2AC2OKPWBMQ, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:12.731693027 (502h58m12.602776062s) [noise=true , address=77.90.31.17:4799 , piece_id=NG3PPYMUUODKZVAVYKGTWZ2QBHOHINP45ZAMOBTQBZBJ6VOPPFNA, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:12.732138174 (502h58m12.603221209s) [noise=true , address=77.90.31.13:4640 , piece_id=J2UOIRDKCGLOS6FPEQCM5SZWGPDPRPO6O2HFY2KJMSWTFZEPRAMQ, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:12.732479955 (502h58m12.603563s) [noise=true , address=64.49.13.27:4957 , piece_id=SI7ANTUA53SPUXSLY2ECQUDHIHWHPHMFTPFWNBMANKZGUW4U2L4A, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:12.732773655 (502h58m12.60385669s) [noise=true , address=77.90.31.18:4832 , piece_id=XZSYVBXPVPA73GHQBP7PFICVBSO3RNC3YLIOSYGIPBKQ66FHQPWA, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:12.733031345 (502h58m12.60411439s) [noise=true , address=64.49.13.169:4711 , piece_id=Q2LQKJ2RBKW2QLHGWIVPWJKBCUQZBQFFOPJGEWMPZZWAILDUJBUQ, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:12.733433241 (502h58m12.604516286s) [noise=true , address=77.90.31.12:4614 , piece_id=5CPNE3UG2AWMOSC2MA6VOSJQLUUDENYYBUQGFHXYN3ON63PVM7EQ, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:12.733723934 (502h58m12.604806969s) [noise=true , address=77.90.31.16:4753 , piece_id=2GMK4UAMU6AHDTQEW447NXYGDKKSZOUFJ7PGL5OGJ5W2GIECTQMQ, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:12.744912929 (502h58m12.615995964s) [noise=true , address=145.79.227.33:4884 , piece_id=JDO3F6HQRBGHWJ74ZVJNML2X5757PP55DWQLF4YT3P7JDKZ7VZIA, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:12.748930370 (502h58m12.620013415s) [noise=true , address=145.79.227.30:4772 , piece_id=MBZYVHFEAHN2FEVS6LHPUCCBHX74E35JO6EOG4GPKBRG7UIK2IPA, seg_pos={0 0}, upload=441384]: Uploading piece...
Dec 26 14:27:14.090029017 (502h58m13.961112062s) [upload=441384]: Next returned nil segment
Dec 26 14:27:14.090042262 (502h58m13.961125297s) [upload=441384]: Waiting for error group managing segments...
Dec 26 14:27:14.444533658 (502h58m14.315616693s) [noise=true , address=64.49.13.139:4480 , piece_id=KGAH4B57THDLG3FIMSEKYO2PE5JXKYLBLRMQQH37CKR5YCBL3QGQ, seg_pos={0 0}, upload=441384]: Done uploading piece. err: <nil>
Dec 26 14:27:14.586227048 (502h58m14.457310083s) [noise=true , address=64.49.13.27:4957 , piece_id=SI7ANTUA53SPUXSLY2ECQUDHIHWHPHMFTPFWNBMANKZGUW4U2L4A, seg_pos={0 0}, upload=441384]: Done uploading piece. err: <nil>
Dec 26 14:27:14.622921881 (502h58m14.494004916s) [noise=true , address=145.79.227.20:4452 , piece_id=UGIUMXQNQIXGGNF2BOQTADA5JOUQKEI27DTWXFB6DIJUJ2GTGBQQ, seg_pos={0 0}, upload=441384]: Done uploading piece. err: <nil>
Dec 26 14:27:14.812511214 (502h58m14.683594259s) [noise=true , address=77.90.31.21:4909 , piece_id=7GZ6PRPYTSWSCFWRSFIBJ5EKN7FETMBWUYJKSS2YVKG3JJK42S3A, seg_pos={0 0}, upload=441384]: Done uploading piece. err: <nil>
Dec 26 14:27:14.823469320 (502h58m14.694552355s) [noise=true , address=77.90.31.18:4832 , piece_id=XZSYVBXPVPA73GHQBP7PFICVBSO3RNC3YLIOSYGIPBKQ66FHQPWA, seg_pos={0 0}, upload=441384]: Done uploading piece. err: <nil>
Dec 26 14:27:14.884791981 (502h58m14.755875016s) [noise=true , address=77.90.31.15:4725 , piece_id=UAASM7Q6N6SJFL3BZQG4J3PWGICDNX7HWKKMLAE3Z4DTLF2CQAHQ, seg_pos={0 0}, upload=441384]: Done uploading piece. err: <nil>
Dec 26 14:27:14.897429475 (502h58m14.76851251s) [noise=true , address=77.90.31.12:4614 , piece_id=5CPNE3UG2AWMOSC2MA6VOSJQLUUDENYYBUQGFHXYN3ON63PVM7EQ, seg_pos={0 0}, upload=441384]: Done uploading piece. err: <nil>
Dec 26 14:27:14.921617022 (502h58m14.792700057s) [noise=true , address=77.90.31.13:4640 , piece_id=J2UOIRDKCGLOS6FPEQCM5SZWGPDPRPO6O2HFY2KJMSWTFZEPRAMQ, seg_pos={0 0}, upload=441384]: Done uploading piece. err: <nil>
Dec 26 14:27:15.217935085 (502h58m15.089018121s) [noise=true , address=77.90.31.17:4799 , piece_id=NG3PPYMUUODKZVAVYKGTWZ2QBHOHINP45ZAMOBTQBZBJ6VOPPFNA, seg_pos={0 0}, upload=441384]: Done uploading piece. err: <nil>
Dec 26 14:27:15.538396774 (502h58m15.409479819s) [noise=true , address=45.91.116.10:4097 , piece_id=WSRZPIBVTNRPGFTBESN6ZEWLITLZOZLBB4IPRAUPYH3M6YEQOWZQ, seg_pos={0 0}, upload=441384]: Done uploading piece. err: <nil>
Dec 26 14:27:15.575796018 (502h58m15.446879063s) [noise=true , address=145.79.227.34:4934 , piece_id=ATCG3GH3MMQSVE5AN6H4JD7WKXPFAOXIR2YIHTHXY2AC2OKPWBMQ, seg_pos={0 0}, upload=441384]: Done uploading piece. err: <nil>
Dec 26 14:27:15.588172846 (502h58m15.459255881s) [noise=true , address=145.79.227.30:4772 , piece_id=MBZYVHFEAHN2FEVS6LHPUCCBHX74E35JO6EOG4GPKBRG7UIK2IPA, seg_pos={0 0}, upload=441384]: Done uploading piece. err: <nil>
Dec 26 14:27:15.654034369 (502h58m15.525117404s) [noise=true , address=145.79.227.44:4221 , piece_id=KNE5IJVSCFLIOLLXHC32A4X4DMVQKV6WHWKA7NSWSYYC4F7OMOFA, seg_pos={0 0}, upload=441384]: Done uploading piece. err: <nil>
Dec 26 14:27:15.764565211 (502h58m15.635648246s) [noise=true , address=145.79.227.40:4076 , piece_id=XVEBWXMTJ7VK2KCPYXS2LDXP2BRI2JYMV6EZF6XMMVTMQ4ICA2BQ, seg_pos={0 0}, upload=441384]: Done uploading piece. err: <nil>
Dec 26 14:27:15.764584317 (502h58m15.635667352s) [seg_pos={0 0}, upload=441384]: Segment reached optimal threshold of 14 pieces.
Dec 26 14:27:15.764749782 (502h58m15.635832817s) [noise=true , address=64.49.13.169:4711 , piece_id=Q2LQKJ2RBKW2QLHGWIVPWJKBCUQZBQFFOPJGEWMPZZWAILDUJBUQ, seg_pos={0 0}, upload=441384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:36534->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:36534->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36534->64.49.13.169:4711: use of closed network connection
Dec 26 14:27:15.764755412 (502h58m15.635838447s) [noise=true , address=64.49.13.169:4711 , piece_id=Q2LQKJ2RBKW2QLHGWIVPWJKBCUQZBQFFOPJGEWMPZZWAILDUJBUQ, seg_pos={0 0}, upload=441384]: Long tail cancellation due to optimal threshold
Dec 26 14:27:15.764867776 (502h58m15.635950811s) [noise=true , address=45.91.116.12:4236 , piece_id=M5R7IAUXCFF7QF4HHWME6H7PKQJYRKBGLGU5VFJJFQE6LUPIYCJA, seg_pos={0 0}, upload=441384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH): protocol: expected piece hash; context canceled; EOF
Dec 26 14:27:15.764897062 (502h58m15.635980097s) [noise=true , address=145.79.227.33:4884 , piece_id=JDO3F6HQRBGHWJ74ZVJNML2X5757PP55DWQLF4YT3P7JDKZ7VZIA, seg_pos={0 0}, upload=441384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qGXdnm83P4eMnCtZdRt2x5yAWP7pQafWwoPG72daZJVR9pyKG): protocol: expected piece hash; context canceled; EOF
Dec 26 14:27:15.764903855 (502h58m15.63598689s) [noise=true , address=145.79.227.33:4884 , piece_id=JDO3F6HQRBGHWJ74ZVJNML2X5757PP55DWQLF4YT3P7JDKZ7VZIA, seg_pos={0 0}, upload=441384]: Long tail cancellation due to optimal threshold
Dec 26 14:27:15.764910517 (502h58m15.635993552s) [noise=true , address=45.91.116.12:4236 , piece_id=M5R7IAUXCFF7QF4HHWME6H7PKQJYRKBGLGU5VFJJFQE6LUPIYCJA, seg_pos={0 0}, upload=441384]: Long tail cancellation due to optimal threshold
Dec 26 14:27:15.765067747 (502h58m15.636150782s) [noise=true , address=77.90.31.16:4753 , piece_id=2GMK4UAMU6AHDTQEW447NXYGDKKSZOUFJ7PGL5OGJ5W2GIECTQMQ, seg_pos={0 0}, upload=441384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE): protocol: expected piece hash; context canceled; EOF
Dec 26 14:27:15.765076293 (502h58m15.636159328s) [noise=true , address=77.90.31.16:4753 , piece_id=2GMK4UAMU6AHDTQEW447NXYGDKKSZOUFJ7PGL5OGJ5W2GIECTQMQ, seg_pos={0 0}, upload=441384]: Long tail cancellation due to optimal threshold
Dec 26 14:27:15.765245906 (502h58m15.636328941s) [seg_pos={0 0}, upload=441384]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 14:27:15.765317863 (502h58m15.636400908s) [upload=441384]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:27:15.765322372 (502h58m15.636405407s) [upload=441384]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:27:16.068012309 (502h58m15.939095344s) [upload=441384]: Done uploading segments...
Dec 26 14:27:16.068037878 (502h58m15.939120913s) [upload=441384]: Upload finished. err: <nil>
Duration: 56.874550982s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.X.........................................................X] duration:
30.00150543s trace:
https://jaeger.dev.storj.io/trace/3ccdb98964ad100c
FAIL (2025-12-26 14:29:03 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001525207s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
53 out of 60 is failed (threshold=6) [X.XX.XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX.XXXXX.XXXXXXX.XXXXX] duration:
5.000839591s trace:
https://jaeger.dev.storj.io/trace/3d21410f06096d14
FAIL (2025-12-26 14:28:47 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000863779s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
10.705946174s trace:
https://jaeger.dev.storj.io/trace/5b63846e68ea1b18
FAIL (2025-12-26 14:28:39 UTC)
orbiter warning: context deadline exceeded
Duration: 10.706003905s
core
uplink
us-select-1
us1-placement-20
|