Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....X.....................X.................................] duration: 4.948956494s trace: https://jaeger.dev.storj.io/trace/35e59be92effb9d0 FAIL (2025-12-13 11:28:36 UTC) orbiter warning: context deadline exceeded Duration: 4.949019274s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 7.69550938s trace: https://jaeger.dev.storj.io/trace/7a99583e03db57d FAIL (2025-12-13 11:28:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ) context deadline exceeded; piecestore: (Node ID: 12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, Piece ID: EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA) context deadline exceeded; piecestore: (Node ID: 12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, Piece ID: AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ) context deadline exceeded ======LOGS====== Dec 13 10:54:04.678153217 (187h25m4.549236252s) [upload=164140]: Uploading segments... Dec 13 10:54:04.678304274 (187h25m4.549387319s) [upload=164140]: Got next segment. Inline: false Dec 13 10:54:04.678334932 (187h25m4.549417967s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 13 10:54:04.678339972 (187h25m4.549423017s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 13 10:54:06.552039235 (187h25m6.42312228s) [seg_pos={0 0}, upload=164140]: Begin upload segment... Dec 13 10:54:06.552085274 (187h25m6.423168309s) [seg_pos={0 0}, upload=164140]: Done begin upload segment. Dec 13 10:54:06.552111302 (187h25m6.423194338s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552155768 (187h25m6.423238813s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552846924 (187h25m6.423929959s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553257215 (187h25m6.42434026s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553392723 (187h25m6.424475758s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553588406 (187h25m6.424671451s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553840846 (187h25m6.424923891s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553974851 (187h25m6.425057897s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554228795 (187h25m6.42531184s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554359133 (187h25m6.425442178s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554712255 (187h25m6.4257953s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554907297 (187h25m6.425990342s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555042745 (187h25m6.42612579s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555365890 (187h25m6.426448925s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555459569 (187h25m6.426542614s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555696730 (187h25m6.426779765s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:08.379289923 (187h25m8.250372968s) [upload=164140]: Next returned nil segment Dec 13 10:54:08.379316804 (187h25m8.25039985s) [upload=164140]: Waiting for error group managing segments... Dec 13 10:54:08.687944366 (187h25m8.559027411s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.688191346 (187h25m8.559274391s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.695282860 (187h25m8.566365905s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.699745637 (187h25m8.570828682s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719582961 (187h25m8.590666006s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719768304 (187h25m8.590851349s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720334432 (187h25m8.591417477s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720634563 (187h25m8.591717609s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.724972483 (187h25m8.596055528s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.739884178 (187h25m8.610967223s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.876599508 (187h25m8.747682553s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.943217551 (187h25m8.814300586s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000359268 (187h25m8.871442323s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000386250 (187h25m8.871469295s) [seg_pos={0 0}, upload=164140]: Segment reached optimal threshold of 13 pieces. Dec 13 10:54:09.000658989 (187h25m8.871742094s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection Dec 13 10:54:09.000669228 (187h25m8.871752273s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.000925507 (187h25m8.872008542s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: read tcp 188.166.225.170:42076->145.79.227.48:4302: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.000933592 (187h25m8.872016637s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001235117 (187h25m8.872318172s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: read tcp 188.166.225.170:36732->145.79.227.50:4374: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.001262077 (187h25m8.872345142s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001292736 (187h25m8.872375791s) [seg_pos={0 0}, upload=164140]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 13 10:54:09.001392175 (187h25m8.87247522s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 10:54:09.001399449 (187h25m8.872482494s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 10:54:09.331221628 (187h25m9.202304674s) [upload=164140]: Done uploading segments... Dec 13 10:54:09.331245344 (187h25m9.202328389s) [upload=164140]: Upload finished. err: <nil> Duration: 7.695539347s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....X.....................X................................] duration: 3.533516966s trace: https://jaeger.dev.storj.io/trace/35e59be92effb9d0 FAIL (2025-12-13 11:27:43 UTC) orbiter warning: context deadline exceeded Duration: 3.533542826s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.656637062s trace: https://jaeger.dev.storj.io/trace/7a99583e03db57d FAIL (2025-12-13 11:27:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ) context deadline exceeded; piecestore: (Node ID: 12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, Piece ID: EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA) context deadline exceeded; piecestore: (Node ID: 12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, Piece ID: AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ) context deadline exceeded ======LOGS====== Dec 13 10:54:04.678153217 (187h25m4.549236252s) [upload=164140]: Uploading segments... Dec 13 10:54:04.678304274 (187h25m4.549387319s) [upload=164140]: Got next segment. Inline: false Dec 13 10:54:04.678334932 (187h25m4.549417967s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 13 10:54:04.678339972 (187h25m4.549423017s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 13 10:54:06.552039235 (187h25m6.42312228s) [seg_pos={0 0}, upload=164140]: Begin upload segment... Dec 13 10:54:06.552085274 (187h25m6.423168309s) [seg_pos={0 0}, upload=164140]: Done begin upload segment. Dec 13 10:54:06.552111302 (187h25m6.423194338s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552155768 (187h25m6.423238813s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552846924 (187h25m6.423929959s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553257215 (187h25m6.42434026s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553392723 (187h25m6.424475758s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553588406 (187h25m6.424671451s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553840846 (187h25m6.424923891s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553974851 (187h25m6.425057897s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554228795 (187h25m6.42531184s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554359133 (187h25m6.425442178s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554712255 (187h25m6.4257953s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554907297 (187h25m6.425990342s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555042745 (187h25m6.42612579s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555365890 (187h25m6.426448925s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555459569 (187h25m6.426542614s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555696730 (187h25m6.426779765s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:08.379289923 (187h25m8.250372968s) [upload=164140]: Next returned nil segment Dec 13 10:54:08.379316804 (187h25m8.25039985s) [upload=164140]: Waiting for error group managing segments... Dec 13 10:54:08.687944366 (187h25m8.559027411s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.688191346 (187h25m8.559274391s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.695282860 (187h25m8.566365905s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.699745637 (187h25m8.570828682s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719582961 (187h25m8.590666006s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719768304 (187h25m8.590851349s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720334432 (187h25m8.591417477s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720634563 (187h25m8.591717609s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.724972483 (187h25m8.596055528s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.739884178 (187h25m8.610967223s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.876599508 (187h25m8.747682553s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.943217551 (187h25m8.814300586s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000359268 (187h25m8.871442323s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000386250 (187h25m8.871469295s) [seg_pos={0 0}, upload=164140]: Segment reached optimal threshold of 13 pieces. Dec 13 10:54:09.000658989 (187h25m8.871742094s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection Dec 13 10:54:09.000669228 (187h25m8.871752273s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.000925507 (187h25m8.872008542s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: read tcp 188.166.225.170:42076->145.79.227.48:4302: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.000933592 (187h25m8.872016637s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001235117 (187h25m8.872318172s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: read tcp 188.166.225.170:36732->145.79.227.50:4374: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.001262077 (187h25m8.872345142s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001292736 (187h25m8.872375791s) [seg_pos={0 0}, upload=164140]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 13 10:54:09.001392175 (187h25m8.87247522s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 10:54:09.001399449 (187h25m8.872482494s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 10:54:09.331221628 (187h25m9.202304674s) [upload=164140]: Done uploading segments... Dec 13 10:54:09.331245344 (187h25m9.202328389s) [upload=164140]: Upload finished. err: <nil> Duration: 6.656664943s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.47856932s trace: https://jaeger.dev.storj.io/trace/7a99583e03db57d FAIL (2025-12-13 11:27:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ) context deadline exceeded; piecestore: (Node ID: 12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, Piece ID: EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA) context deadline exceeded; piecestore: (Node ID: 12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, Piece ID: AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ) context deadline exceeded ======LOGS====== Dec 13 10:54:04.678153217 (187h25m4.549236252s) [upload=164140]: Uploading segments... Dec 13 10:54:04.678304274 (187h25m4.549387319s) [upload=164140]: Got next segment. Inline: false Dec 13 10:54:04.678334932 (187h25m4.549417967s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 13 10:54:04.678339972 (187h25m4.549423017s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 13 10:54:06.552039235 (187h25m6.42312228s) [seg_pos={0 0}, upload=164140]: Begin upload segment... Dec 13 10:54:06.552085274 (187h25m6.423168309s) [seg_pos={0 0}, upload=164140]: Done begin upload segment. Dec 13 10:54:06.552111302 (187h25m6.423194338s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552155768 (187h25m6.423238813s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552846924 (187h25m6.423929959s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553257215 (187h25m6.42434026s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553392723 (187h25m6.424475758s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553588406 (187h25m6.424671451s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553840846 (187h25m6.424923891s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553974851 (187h25m6.425057897s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554228795 (187h25m6.42531184s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554359133 (187h25m6.425442178s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554712255 (187h25m6.4257953s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554907297 (187h25m6.425990342s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555042745 (187h25m6.42612579s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555365890 (187h25m6.426448925s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555459569 (187h25m6.426542614s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555696730 (187h25m6.426779765s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:08.379289923 (187h25m8.250372968s) [upload=164140]: Next returned nil segment Dec 13 10:54:08.379316804 (187h25m8.25039985s) [upload=164140]: Waiting for error group managing segments... Dec 13 10:54:08.687944366 (187h25m8.559027411s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.688191346 (187h25m8.559274391s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.695282860 (187h25m8.566365905s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.699745637 (187h25m8.570828682s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719582961 (187h25m8.590666006s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719768304 (187h25m8.590851349s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720334432 (187h25m8.591417477s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720634563 (187h25m8.591717609s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.724972483 (187h25m8.596055528s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.739884178 (187h25m8.610967223s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.876599508 (187h25m8.747682553s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.943217551 (187h25m8.814300586s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000359268 (187h25m8.871442323s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000386250 (187h25m8.871469295s) [seg_pos={0 0}, upload=164140]: Segment reached optimal threshold of 13 pieces. Dec 13 10:54:09.000658989 (187h25m8.871742094s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection Dec 13 10:54:09.000669228 (187h25m8.871752273s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.000925507 (187h25m8.872008542s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: read tcp 188.166.225.170:42076->145.79.227.48:4302: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.000933592 (187h25m8.872016637s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001235117 (187h25m8.872318172s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: read tcp 188.166.225.170:36732->145.79.227.50:4374: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.001262077 (187h25m8.872345142s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001292736 (187h25m8.872375791s) [seg_pos={0 0}, upload=164140]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 13 10:54:09.001392175 (187h25m8.87247522s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 10:54:09.001399449 (187h25m8.872482494s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 10:54:09.331221628 (187h25m9.202304674s) [upload=164140]: Done uploading segments... Dec 13 10:54:09.331245344 (187h25m9.202328389s) [upload=164140]: Upload finished. err: <nil> Duration: 6.478592895s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......X.....................X...............................] duration: 3.556509046s trace: https://jaeger.dev.storj.io/trace/35e59be92effb9d0 FAIL (2025-12-13 11:27:06 UTC) orbiter warning: context deadline exceeded Duration: 3.556545025s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 7.701378727s trace: https://jaeger.dev.storj.io/trace/7a99583e03db57d FAIL (2025-12-13 11:26:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ) context deadline exceeded; piecestore: (Node ID: 12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, Piece ID: EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA) context deadline exceeded; piecestore: (Node ID: 12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, Piece ID: AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ) context deadline exceeded ======LOGS====== Dec 13 10:54:04.678153217 (187h25m4.549236252s) [upload=164140]: Uploading segments... Dec 13 10:54:04.678304274 (187h25m4.549387319s) [upload=164140]: Got next segment. Inline: false Dec 13 10:54:04.678334932 (187h25m4.549417967s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 13 10:54:04.678339972 (187h25m4.549423017s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 13 10:54:06.552039235 (187h25m6.42312228s) [seg_pos={0 0}, upload=164140]: Begin upload segment... Dec 13 10:54:06.552085274 (187h25m6.423168309s) [seg_pos={0 0}, upload=164140]: Done begin upload segment. Dec 13 10:54:06.552111302 (187h25m6.423194338s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552155768 (187h25m6.423238813s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552846924 (187h25m6.423929959s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553257215 (187h25m6.42434026s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553392723 (187h25m6.424475758s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553588406 (187h25m6.424671451s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553840846 (187h25m6.424923891s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553974851 (187h25m6.425057897s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554228795 (187h25m6.42531184s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554359133 (187h25m6.425442178s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554712255 (187h25m6.4257953s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554907297 (187h25m6.425990342s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555042745 (187h25m6.42612579s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555365890 (187h25m6.426448925s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555459569 (187h25m6.426542614s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555696730 (187h25m6.426779765s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:08.379289923 (187h25m8.250372968s) [upload=164140]: Next returned nil segment Dec 13 10:54:08.379316804 (187h25m8.25039985s) [upload=164140]: Waiting for error group managing segments... Dec 13 10:54:08.687944366 (187h25m8.559027411s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.688191346 (187h25m8.559274391s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.695282860 (187h25m8.566365905s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.699745637 (187h25m8.570828682s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719582961 (187h25m8.590666006s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719768304 (187h25m8.590851349s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720334432 (187h25m8.591417477s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720634563 (187h25m8.591717609s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.724972483 (187h25m8.596055528s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.739884178 (187h25m8.610967223s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.876599508 (187h25m8.747682553s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.943217551 (187h25m8.814300586s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000359268 (187h25m8.871442323s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000386250 (187h25m8.871469295s) [seg_pos={0 0}, upload=164140]: Segment reached optimal threshold of 13 pieces. Dec 13 10:54:09.000658989 (187h25m8.871742094s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection Dec 13 10:54:09.000669228 (187h25m8.871752273s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.000925507 (187h25m8.872008542s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: read tcp 188.166.225.170:42076->145.79.227.48:4302: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.000933592 (187h25m8.872016637s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001235117 (187h25m8.872318172s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: read tcp 188.166.225.170:36732->145.79.227.50:4374: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.001262077 (187h25m8.872345142s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001292736 (187h25m8.872375791s) [seg_pos={0 0}, upload=164140]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 13 10:54:09.001392175 (187h25m8.87247522s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 10:54:09.001399449 (187h25m8.872482494s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 10:54:09.331221628 (187h25m9.202304674s) [upload=164140]: Done uploading segments... Dec 13 10:54:09.331245344 (187h25m9.202328389s) [upload=164140]: Upload finished. err: <nil> Duration: 7.701405418s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 9.021841103s trace: https://jaeger.dev.storj.io/trace/7a99583e03db57d FAIL (2025-12-13 11:26:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ) context deadline exceeded; piecestore: (Node ID: 12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, Piece ID: EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA) context deadline exceeded; piecestore: (Node ID: 12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, Piece ID: AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ) context deadline exceeded ======LOGS====== Dec 13 10:54:04.678153217 (187h25m4.549236252s) [upload=164140]: Uploading segments... Dec 13 10:54:04.678304274 (187h25m4.549387319s) [upload=164140]: Got next segment. Inline: false Dec 13 10:54:04.678334932 (187h25m4.549417967s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 13 10:54:04.678339972 (187h25m4.549423017s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 13 10:54:06.552039235 (187h25m6.42312228s) [seg_pos={0 0}, upload=164140]: Begin upload segment... Dec 13 10:54:06.552085274 (187h25m6.423168309s) [seg_pos={0 0}, upload=164140]: Done begin upload segment. Dec 13 10:54:06.552111302 (187h25m6.423194338s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552155768 (187h25m6.423238813s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552846924 (187h25m6.423929959s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553257215 (187h25m6.42434026s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553392723 (187h25m6.424475758s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553588406 (187h25m6.424671451s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553840846 (187h25m6.424923891s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553974851 (187h25m6.425057897s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554228795 (187h25m6.42531184s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554359133 (187h25m6.425442178s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554712255 (187h25m6.4257953s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554907297 (187h25m6.425990342s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555042745 (187h25m6.42612579s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555365890 (187h25m6.426448925s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555459569 (187h25m6.426542614s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555696730 (187h25m6.426779765s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:08.379289923 (187h25m8.250372968s) [upload=164140]: Next returned nil segment Dec 13 10:54:08.379316804 (187h25m8.25039985s) [upload=164140]: Waiting for error group managing segments... Dec 13 10:54:08.687944366 (187h25m8.559027411s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.688191346 (187h25m8.559274391s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.695282860 (187h25m8.566365905s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.699745637 (187h25m8.570828682s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719582961 (187h25m8.590666006s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719768304 (187h25m8.590851349s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720334432 (187h25m8.591417477s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720634563 (187h25m8.591717609s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.724972483 (187h25m8.596055528s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.739884178 (187h25m8.610967223s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.876599508 (187h25m8.747682553s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.943217551 (187h25m8.814300586s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000359268 (187h25m8.871442323s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000386250 (187h25m8.871469295s) [seg_pos={0 0}, upload=164140]: Segment reached optimal threshold of 13 pieces. Dec 13 10:54:09.000658989 (187h25m8.871742094s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection Dec 13 10:54:09.000669228 (187h25m8.871752273s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.000925507 (187h25m8.872008542s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: read tcp 188.166.225.170:42076->145.79.227.48:4302: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.000933592 (187h25m8.872016637s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001235117 (187h25m8.872318172s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: read tcp 188.166.225.170:36732->145.79.227.50:4374: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.001262077 (187h25m8.872345142s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001292736 (187h25m8.872375791s) [seg_pos={0 0}, upload=164140]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 13 10:54:09.001392175 (187h25m8.87247522s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 10:54:09.001399449 (187h25m8.872482494s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 10:54:09.331221628 (187h25m9.202304674s) [upload=164140]: Done uploading segments... Dec 13 10:54:09.331245344 (187h25m9.202328389s) [upload=164140]: Upload finished. err: <nil> Duration: 9.021870721s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......X.....................X..............................] duration: 3.508456847s trace: https://jaeger.dev.storj.io/trace/35e59be92effb9d0 FAIL (2025-12-13 11:25:57 UTC) orbiter warning: context deadline exceeded Duration: 3.508481765s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 7.021917774s trace: https://jaeger.dev.storj.io/trace/7a99583e03db57d FAIL (2025-12-13 11:25:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ) context deadline exceeded; piecestore: (Node ID: 12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, Piece ID: EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA) context deadline exceeded; piecestore: (Node ID: 12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, Piece ID: AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ) context deadline exceeded ======LOGS====== Dec 13 10:54:04.678153217 (187h25m4.549236252s) [upload=164140]: Uploading segments... Dec 13 10:54:04.678304274 (187h25m4.549387319s) [upload=164140]: Got next segment. Inline: false Dec 13 10:54:04.678334932 (187h25m4.549417967s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 13 10:54:04.678339972 (187h25m4.549423017s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 13 10:54:06.552039235 (187h25m6.42312228s) [seg_pos={0 0}, upload=164140]: Begin upload segment... Dec 13 10:54:06.552085274 (187h25m6.423168309s) [seg_pos={0 0}, upload=164140]: Done begin upload segment. Dec 13 10:54:06.552111302 (187h25m6.423194338s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552155768 (187h25m6.423238813s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552846924 (187h25m6.423929959s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553257215 (187h25m6.42434026s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553392723 (187h25m6.424475758s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553588406 (187h25m6.424671451s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553840846 (187h25m6.424923891s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553974851 (187h25m6.425057897s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554228795 (187h25m6.42531184s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554359133 (187h25m6.425442178s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554712255 (187h25m6.4257953s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554907297 (187h25m6.425990342s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555042745 (187h25m6.42612579s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555365890 (187h25m6.426448925s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555459569 (187h25m6.426542614s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555696730 (187h25m6.426779765s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:08.379289923 (187h25m8.250372968s) [upload=164140]: Next returned nil segment Dec 13 10:54:08.379316804 (187h25m8.25039985s) [upload=164140]: Waiting for error group managing segments... Dec 13 10:54:08.687944366 (187h25m8.559027411s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.688191346 (187h25m8.559274391s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.695282860 (187h25m8.566365905s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.699745637 (187h25m8.570828682s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719582961 (187h25m8.590666006s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719768304 (187h25m8.590851349s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720334432 (187h25m8.591417477s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720634563 (187h25m8.591717609s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.724972483 (187h25m8.596055528s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.739884178 (187h25m8.610967223s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.876599508 (187h25m8.747682553s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.943217551 (187h25m8.814300586s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000359268 (187h25m8.871442323s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000386250 (187h25m8.871469295s) [seg_pos={0 0}, upload=164140]: Segment reached optimal threshold of 13 pieces. Dec 13 10:54:09.000658989 (187h25m8.871742094s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection Dec 13 10:54:09.000669228 (187h25m8.871752273s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.000925507 (187h25m8.872008542s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: read tcp 188.166.225.170:42076->145.79.227.48:4302: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.000933592 (187h25m8.872016637s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001235117 (187h25m8.872318172s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: read tcp 188.166.225.170:36732->145.79.227.50:4374: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.001262077 (187h25m8.872345142s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001292736 (187h25m8.872375791s) [seg_pos={0 0}, upload=164140]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 13 10:54:09.001392175 (187h25m8.87247522s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 10:54:09.001399449 (187h25m8.872482494s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 10:54:09.331221628 (187h25m9.202304674s) [upload=164140]: Done uploading segments... Dec 13 10:54:09.331245344 (187h25m9.202328389s) [upload=164140]: Upload finished. err: <nil> Duration: 7.021941658s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........X.....................X.............................] duration: 3.494419809s trace: https://jaeger.dev.storj.io/trace/35e59be92effb9d0 FAIL (2025-12-13 11:24:56 UTC) orbiter warning: context deadline exceeded Duration: 3.494497256s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 8.610130291s trace: https://jaeger.dev.storj.io/trace/7a99583e03db57d FAIL (2025-12-13 11:24:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ) context deadline exceeded; piecestore: (Node ID: 12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, Piece ID: EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA) context deadline exceeded; piecestore: (Node ID: 12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, Piece ID: AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ) context deadline exceeded ======LOGS====== Dec 13 10:54:04.678153217 (187h25m4.549236252s) [upload=164140]: Uploading segments... Dec 13 10:54:04.678304274 (187h25m4.549387319s) [upload=164140]: Got next segment. Inline: false Dec 13 10:54:04.678334932 (187h25m4.549417967s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 13 10:54:04.678339972 (187h25m4.549423017s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 13 10:54:06.552039235 (187h25m6.42312228s) [seg_pos={0 0}, upload=164140]: Begin upload segment... Dec 13 10:54:06.552085274 (187h25m6.423168309s) [seg_pos={0 0}, upload=164140]: Done begin upload segment. Dec 13 10:54:06.552111302 (187h25m6.423194338s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552155768 (187h25m6.423238813s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552846924 (187h25m6.423929959s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553257215 (187h25m6.42434026s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553392723 (187h25m6.424475758s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553588406 (187h25m6.424671451s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553840846 (187h25m6.424923891s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553974851 (187h25m6.425057897s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554228795 (187h25m6.42531184s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554359133 (187h25m6.425442178s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554712255 (187h25m6.4257953s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554907297 (187h25m6.425990342s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555042745 (187h25m6.42612579s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555365890 (187h25m6.426448925s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555459569 (187h25m6.426542614s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555696730 (187h25m6.426779765s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:08.379289923 (187h25m8.250372968s) [upload=164140]: Next returned nil segment Dec 13 10:54:08.379316804 (187h25m8.25039985s) [upload=164140]: Waiting for error group managing segments... Dec 13 10:54:08.687944366 (187h25m8.559027411s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.688191346 (187h25m8.559274391s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.695282860 (187h25m8.566365905s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.699745637 (187h25m8.570828682s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719582961 (187h25m8.590666006s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719768304 (187h25m8.590851349s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720334432 (187h25m8.591417477s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720634563 (187h25m8.591717609s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.724972483 (187h25m8.596055528s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.739884178 (187h25m8.610967223s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.876599508 (187h25m8.747682553s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.943217551 (187h25m8.814300586s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000359268 (187h25m8.871442323s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000386250 (187h25m8.871469295s) [seg_pos={0 0}, upload=164140]: Segment reached optimal threshold of 13 pieces. Dec 13 10:54:09.000658989 (187h25m8.871742094s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection Dec 13 10:54:09.000669228 (187h25m8.871752273s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.000925507 (187h25m8.872008542s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: read tcp 188.166.225.170:42076->145.79.227.48:4302: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.000933592 (187h25m8.872016637s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001235117 (187h25m8.872318172s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: read tcp 188.166.225.170:36732->145.79.227.50:4374: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.001262077 (187h25m8.872345142s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001292736 (187h25m8.872375791s) [seg_pos={0 0}, upload=164140]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 13 10:54:09.001392175 (187h25m8.87247522s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 10:54:09.001399449 (187h25m8.872482494s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 10:54:09.331221628 (187h25m9.202304674s) [upload=164140]: Done uploading segments... Dec 13 10:54:09.331245344 (187h25m9.202328389s) [upload=164140]: Upload finished. err: <nil> Duration: 8.610159266s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........X.....................X............................] duration: 3.4733416s trace: https://jaeger.dev.storj.io/trace/35e59be92effb9d0 FAIL (2025-12-13 11:24:22 UTC) orbiter warning: context deadline exceeded Duration: 3.473370447s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........X.....................X...........................] duration: 3.725352975s trace: https://jaeger.dev.storj.io/trace/35e59be92effb9d0 FAIL (2025-12-13 11:24:06 UTC) orbiter warning: context deadline exceeded Duration: 3.725493101s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........X.....................X..........................] duration: 3.520746618s trace: https://jaeger.dev.storj.io/trace/35e59be92effb9d0 FAIL (2025-12-13 11:23:40 UTC) orbiter warning: context deadline exceeded Duration: 3.520775343s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 8.434497654s trace: https://jaeger.dev.storj.io/trace/7a99583e03db57d FAIL (2025-12-13 11:23:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ) context deadline exceeded; piecestore: (Node ID: 12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, Piece ID: EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA) context deadline exceeded; piecestore: (Node ID: 12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, Piece ID: AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ) context deadline exceeded ======LOGS====== Dec 13 10:54:04.678153217 (187h25m4.549236252s) [upload=164140]: Uploading segments... Dec 13 10:54:04.678304274 (187h25m4.549387319s) [upload=164140]: Got next segment. Inline: false Dec 13 10:54:04.678334932 (187h25m4.549417967s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 13 10:54:04.678339972 (187h25m4.549423017s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 13 10:54:06.552039235 (187h25m6.42312228s) [seg_pos={0 0}, upload=164140]: Begin upload segment... Dec 13 10:54:06.552085274 (187h25m6.423168309s) [seg_pos={0 0}, upload=164140]: Done begin upload segment. Dec 13 10:54:06.552111302 (187h25m6.423194338s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552155768 (187h25m6.423238813s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552846924 (187h25m6.423929959s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553257215 (187h25m6.42434026s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553392723 (187h25m6.424475758s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553588406 (187h25m6.424671451s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553840846 (187h25m6.424923891s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553974851 (187h25m6.425057897s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554228795 (187h25m6.42531184s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554359133 (187h25m6.425442178s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554712255 (187h25m6.4257953s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554907297 (187h25m6.425990342s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555042745 (187h25m6.42612579s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555365890 (187h25m6.426448925s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555459569 (187h25m6.426542614s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555696730 (187h25m6.426779765s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:08.379289923 (187h25m8.250372968s) [upload=164140]: Next returned nil segment Dec 13 10:54:08.379316804 (187h25m8.25039985s) [upload=164140]: Waiting for error group managing segments... Dec 13 10:54:08.687944366 (187h25m8.559027411s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.688191346 (187h25m8.559274391s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.695282860 (187h25m8.566365905s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.699745637 (187h25m8.570828682s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719582961 (187h25m8.590666006s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719768304 (187h25m8.590851349s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720334432 (187h25m8.591417477s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720634563 (187h25m8.591717609s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.724972483 (187h25m8.596055528s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.739884178 (187h25m8.610967223s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.876599508 (187h25m8.747682553s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.943217551 (187h25m8.814300586s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000359268 (187h25m8.871442323s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000386250 (187h25m8.871469295s) [seg_pos={0 0}, upload=164140]: Segment reached optimal threshold of 13 pieces. Dec 13 10:54:09.000658989 (187h25m8.871742094s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection Dec 13 10:54:09.000669228 (187h25m8.871752273s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.000925507 (187h25m8.872008542s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: read tcp 188.166.225.170:42076->145.79.227.48:4302: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.000933592 (187h25m8.872016637s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001235117 (187h25m8.872318172s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: read tcp 188.166.225.170:36732->145.79.227.50:4374: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.001262077 (187h25m8.872345142s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001292736 (187h25m8.872375791s) [seg_pos={0 0}, upload=164140]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 13 10:54:09.001392175 (187h25m8.87247522s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 10:54:09.001399449 (187h25m8.872482494s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 10:54:09.331221628 (187h25m9.202304674s) [upload=164140]: Done uploading segments... Dec 13 10:54:09.331245344 (187h25m9.202328389s) [upload=164140]: Upload finished. err: <nil> Duration: 8.434531768s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............X.....................X.........................] duration: 3.908139904s trace: https://jaeger.dev.storj.io/trace/35e59be92effb9d0 FAIL (2025-12-13 11:23:17 UTC) orbiter warning: context deadline exceeded Duration: 3.908194367s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............X.....................X........................] duration: 3.615688365s trace: https://jaeger.dev.storj.io/trace/35e59be92effb9d0 FAIL (2025-12-13 11:22:44 UTC) orbiter warning: context deadline exceeded Duration: 3.615711689s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 9.036040982s trace: https://jaeger.dev.storj.io/trace/7a99583e03db57d FAIL (2025-12-13 11:22:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ) context deadline exceeded; piecestore: (Node ID: 12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, Piece ID: EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA) context deadline exceeded; piecestore: (Node ID: 12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, Piece ID: AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ) context deadline exceeded ======LOGS====== Dec 13 10:54:04.678153217 (187h25m4.549236252s) [upload=164140]: Uploading segments... Dec 13 10:54:04.678304274 (187h25m4.549387319s) [upload=164140]: Got next segment. Inline: false Dec 13 10:54:04.678334932 (187h25m4.549417967s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 13 10:54:04.678339972 (187h25m4.549423017s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 13 10:54:06.552039235 (187h25m6.42312228s) [seg_pos={0 0}, upload=164140]: Begin upload segment... Dec 13 10:54:06.552085274 (187h25m6.423168309s) [seg_pos={0 0}, upload=164140]: Done begin upload segment. Dec 13 10:54:06.552111302 (187h25m6.423194338s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552155768 (187h25m6.423238813s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552846924 (187h25m6.423929959s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553257215 (187h25m6.42434026s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553392723 (187h25m6.424475758s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553588406 (187h25m6.424671451s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553840846 (187h25m6.424923891s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553974851 (187h25m6.425057897s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554228795 (187h25m6.42531184s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554359133 (187h25m6.425442178s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554712255 (187h25m6.4257953s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554907297 (187h25m6.425990342s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555042745 (187h25m6.42612579s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555365890 (187h25m6.426448925s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555459569 (187h25m6.426542614s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555696730 (187h25m6.426779765s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:08.379289923 (187h25m8.250372968s) [upload=164140]: Next returned nil segment Dec 13 10:54:08.379316804 (187h25m8.25039985s) [upload=164140]: Waiting for error group managing segments... Dec 13 10:54:08.687944366 (187h25m8.559027411s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.688191346 (187h25m8.559274391s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.695282860 (187h25m8.566365905s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.699745637 (187h25m8.570828682s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719582961 (187h25m8.590666006s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719768304 (187h25m8.590851349s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720334432 (187h25m8.591417477s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720634563 (187h25m8.591717609s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.724972483 (187h25m8.596055528s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.739884178 (187h25m8.610967223s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.876599508 (187h25m8.747682553s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.943217551 (187h25m8.814300586s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000359268 (187h25m8.871442323s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000386250 (187h25m8.871469295s) [seg_pos={0 0}, upload=164140]: Segment reached optimal threshold of 13 pieces. Dec 13 10:54:09.000658989 (187h25m8.871742094s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection Dec 13 10:54:09.000669228 (187h25m8.871752273s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.000925507 (187h25m8.872008542s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: read tcp 188.166.225.170:42076->145.79.227.48:4302: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.000933592 (187h25m8.872016637s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001235117 (187h25m8.872318172s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: read tcp 188.166.225.170:36732->145.79.227.50:4374: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.001262077 (187h25m8.872345142s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001292736 (187h25m8.872375791s) [seg_pos={0 0}, upload=164140]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 13 10:54:09.001392175 (187h25m8.87247522s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 10:54:09.001399449 (187h25m8.872482494s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 10:54:09.331221628 (187h25m9.202304674s) [upload=164140]: Done uploading segments... Dec 13 10:54:09.331245344 (187h25m9.202328389s) [upload=164140]: Upload finished. err: <nil> Duration: 9.036063785s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............X.....................X.......................] duration: 3.956949628s trace: https://jaeger.dev.storj.io/trace/35e59be92effb9d0 FAIL (2025-12-13 11:21:51 UTC) orbiter warning: context deadline exceeded Duration: 3.956975126s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 8.116288319s trace: https://jaeger.dev.storj.io/trace/7a99583e03db57d FAIL (2025-12-13 11:21:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ) context deadline exceeded; piecestore: (Node ID: 12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, Piece ID: EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA) context deadline exceeded; piecestore: (Node ID: 12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, Piece ID: AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ) context deadline exceeded ======LOGS====== Dec 13 10:54:04.678153217 (187h25m4.549236252s) [upload=164140]: Uploading segments... Dec 13 10:54:04.678304274 (187h25m4.549387319s) [upload=164140]: Got next segment. Inline: false Dec 13 10:54:04.678334932 (187h25m4.549417967s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 13 10:54:04.678339972 (187h25m4.549423017s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 13 10:54:06.552039235 (187h25m6.42312228s) [seg_pos={0 0}, upload=164140]: Begin upload segment... Dec 13 10:54:06.552085274 (187h25m6.423168309s) [seg_pos={0 0}, upload=164140]: Done begin upload segment. Dec 13 10:54:06.552111302 (187h25m6.423194338s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552155768 (187h25m6.423238813s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552846924 (187h25m6.423929959s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553257215 (187h25m6.42434026s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553392723 (187h25m6.424475758s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553588406 (187h25m6.424671451s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553840846 (187h25m6.424923891s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553974851 (187h25m6.425057897s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554228795 (187h25m6.42531184s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554359133 (187h25m6.425442178s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554712255 (187h25m6.4257953s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554907297 (187h25m6.425990342s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555042745 (187h25m6.42612579s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555365890 (187h25m6.426448925s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555459569 (187h25m6.426542614s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555696730 (187h25m6.426779765s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:08.379289923 (187h25m8.250372968s) [upload=164140]: Next returned nil segment Dec 13 10:54:08.379316804 (187h25m8.25039985s) [upload=164140]: Waiting for error group managing segments... Dec 13 10:54:08.687944366 (187h25m8.559027411s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.688191346 (187h25m8.559274391s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.695282860 (187h25m8.566365905s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.699745637 (187h25m8.570828682s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719582961 (187h25m8.590666006s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719768304 (187h25m8.590851349s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720334432 (187h25m8.591417477s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720634563 (187h25m8.591717609s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.724972483 (187h25m8.596055528s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.739884178 (187h25m8.610967223s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.876599508 (187h25m8.747682553s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.943217551 (187h25m8.814300586s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000359268 (187h25m8.871442323s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000386250 (187h25m8.871469295s) [seg_pos={0 0}, upload=164140]: Segment reached optimal threshold of 13 pieces. Dec 13 10:54:09.000658989 (187h25m8.871742094s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection Dec 13 10:54:09.000669228 (187h25m8.871752273s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.000925507 (187h25m8.872008542s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: read tcp 188.166.225.170:42076->145.79.227.48:4302: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.000933592 (187h25m8.872016637s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001235117 (187h25m8.872318172s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: read tcp 188.166.225.170:36732->145.79.227.50:4374: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.001262077 (187h25m8.872345142s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001292736 (187h25m8.872375791s) [seg_pos={0 0}, upload=164140]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 13 10:54:09.001392175 (187h25m8.87247522s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 10:54:09.001399449 (187h25m8.872482494s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 10:54:09.331221628 (187h25m9.202304674s) [upload=164140]: Done uploading segments... Dec 13 10:54:09.331245344 (187h25m9.202328389s) [upload=164140]: Upload finished. err: <nil> Duration: 8.116316853s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 7.189026878s trace: https://jaeger.dev.storj.io/trace/7a99583e03db57d FAIL (2025-12-13 11:20:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ) context deadline exceeded; piecestore: (Node ID: 12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, Piece ID: EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA) context deadline exceeded; piecestore: (Node ID: 12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, Piece ID: AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ) context deadline exceeded ======LOGS====== Dec 13 10:54:04.678153217 (187h25m4.549236252s) [upload=164140]: Uploading segments... Dec 13 10:54:04.678304274 (187h25m4.549387319s) [upload=164140]: Got next segment. Inline: false Dec 13 10:54:04.678334932 (187h25m4.549417967s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 13 10:54:04.678339972 (187h25m4.549423017s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 13 10:54:06.552039235 (187h25m6.42312228s) [seg_pos={0 0}, upload=164140]: Begin upload segment... Dec 13 10:54:06.552085274 (187h25m6.423168309s) [seg_pos={0 0}, upload=164140]: Done begin upload segment. Dec 13 10:54:06.552111302 (187h25m6.423194338s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552155768 (187h25m6.423238813s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552846924 (187h25m6.423929959s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553257215 (187h25m6.42434026s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553392723 (187h25m6.424475758s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553588406 (187h25m6.424671451s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553840846 (187h25m6.424923891s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553974851 (187h25m6.425057897s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554228795 (187h25m6.42531184s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554359133 (187h25m6.425442178s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554712255 (187h25m6.4257953s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554907297 (187h25m6.425990342s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555042745 (187h25m6.42612579s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555365890 (187h25m6.426448925s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555459569 (187h25m6.426542614s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555696730 (187h25m6.426779765s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:08.379289923 (187h25m8.250372968s) [upload=164140]: Next returned nil segment Dec 13 10:54:08.379316804 (187h25m8.25039985s) [upload=164140]: Waiting for error group managing segments... Dec 13 10:54:08.687944366 (187h25m8.559027411s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.688191346 (187h25m8.559274391s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.695282860 (187h25m8.566365905s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.699745637 (187h25m8.570828682s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719582961 (187h25m8.590666006s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719768304 (187h25m8.590851349s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720334432 (187h25m8.591417477s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720634563 (187h25m8.591717609s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.724972483 (187h25m8.596055528s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.739884178 (187h25m8.610967223s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.876599508 (187h25m8.747682553s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.943217551 (187h25m8.814300586s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000359268 (187h25m8.871442323s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000386250 (187h25m8.871469295s) [seg_pos={0 0}, upload=164140]: Segment reached optimal threshold of 13 pieces. Dec 13 10:54:09.000658989 (187h25m8.871742094s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection Dec 13 10:54:09.000669228 (187h25m8.871752273s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.000925507 (187h25m8.872008542s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: read tcp 188.166.225.170:42076->145.79.227.48:4302: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.000933592 (187h25m8.872016637s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001235117 (187h25m8.872318172s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: read tcp 188.166.225.170:36732->145.79.227.50:4374: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.001262077 (187h25m8.872345142s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001292736 (187h25m8.872375791s) [seg_pos={0 0}, upload=164140]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 13 10:54:09.001392175 (187h25m8.87247522s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 10:54:09.001399449 (187h25m8.872482494s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 10:54:09.331221628 (187h25m9.202304674s) [upload=164140]: Done uploading segments... Dec 13 10:54:09.331245344 (187h25m9.202328389s) [upload=164140]: Upload finished. err: <nil> Duration: 7.189048349s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............X.....................X......................] duration: 3.573971429s trace: https://jaeger.dev.storj.io/trace/35e59be92effb9d0 FAIL (2025-12-13 11:20:38 UTC) orbiter warning: context deadline exceeded Duration: 3.57399328s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................X.....................X.....................] duration: 3.498843872s trace: https://jaeger.dev.storj.io/trace/35e59be92effb9d0 FAIL (2025-12-13 11:20:13 UTC) orbiter warning: context deadline exceeded Duration: 3.498876054s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 8.072508664s trace: https://jaeger.dev.storj.io/trace/7a99583e03db57d FAIL (2025-12-13 11:20:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ) context deadline exceeded; piecestore: (Node ID: 12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, Piece ID: EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA) context deadline exceeded; piecestore: (Node ID: 12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, Piece ID: AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ) context deadline exceeded ======LOGS====== Dec 13 10:54:04.678153217 (187h25m4.549236252s) [upload=164140]: Uploading segments... Dec 13 10:54:04.678304274 (187h25m4.549387319s) [upload=164140]: Got next segment. Inline: false Dec 13 10:54:04.678334932 (187h25m4.549417967s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 13 10:54:04.678339972 (187h25m4.549423017s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 13 10:54:06.552039235 (187h25m6.42312228s) [seg_pos={0 0}, upload=164140]: Begin upload segment... Dec 13 10:54:06.552085274 (187h25m6.423168309s) [seg_pos={0 0}, upload=164140]: Done begin upload segment. Dec 13 10:54:06.552111302 (187h25m6.423194338s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552155768 (187h25m6.423238813s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552846924 (187h25m6.423929959s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553257215 (187h25m6.42434026s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553392723 (187h25m6.424475758s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553588406 (187h25m6.424671451s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553840846 (187h25m6.424923891s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553974851 (187h25m6.425057897s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554228795 (187h25m6.42531184s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554359133 (187h25m6.425442178s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554712255 (187h25m6.4257953s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554907297 (187h25m6.425990342s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555042745 (187h25m6.42612579s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555365890 (187h25m6.426448925s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555459569 (187h25m6.426542614s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555696730 (187h25m6.426779765s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:08.379289923 (187h25m8.250372968s) [upload=164140]: Next returned nil segment Dec 13 10:54:08.379316804 (187h25m8.25039985s) [upload=164140]: Waiting for error group managing segments... Dec 13 10:54:08.687944366 (187h25m8.559027411s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.688191346 (187h25m8.559274391s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.695282860 (187h25m8.566365905s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.699745637 (187h25m8.570828682s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719582961 (187h25m8.590666006s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719768304 (187h25m8.590851349s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720334432 (187h25m8.591417477s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720634563 (187h25m8.591717609s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.724972483 (187h25m8.596055528s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.739884178 (187h25m8.610967223s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.876599508 (187h25m8.747682553s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.943217551 (187h25m8.814300586s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000359268 (187h25m8.871442323s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000386250 (187h25m8.871469295s) [seg_pos={0 0}, upload=164140]: Segment reached optimal threshold of 13 pieces. Dec 13 10:54:09.000658989 (187h25m8.871742094s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection Dec 13 10:54:09.000669228 (187h25m8.871752273s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.000925507 (187h25m8.872008542s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: read tcp 188.166.225.170:42076->145.79.227.48:4302: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.000933592 (187h25m8.872016637s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001235117 (187h25m8.872318172s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: read tcp 188.166.225.170:36732->145.79.227.50:4374: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.001262077 (187h25m8.872345142s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001292736 (187h25m8.872375791s) [seg_pos={0 0}, upload=164140]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 13 10:54:09.001392175 (187h25m8.87247522s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 10:54:09.001399449 (187h25m8.872482494s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 10:54:09.331221628 (187h25m9.202304674s) [upload=164140]: Done uploading segments... Dec 13 10:54:09.331245344 (187h25m9.202328389s) [upload=164140]: Upload finished. err: <nil> Duration: 8.072541044s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 7.618202343s trace: https://jaeger.dev.storj.io/trace/7a99583e03db57d FAIL (2025-12-13 11:19:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ) context deadline exceeded; piecestore: (Node ID: 12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, Piece ID: EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA) context deadline exceeded; piecestore: (Node ID: 12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, Piece ID: AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ) context deadline exceeded ======LOGS====== Dec 13 10:54:04.678153217 (187h25m4.549236252s) [upload=164140]: Uploading segments... Dec 13 10:54:04.678304274 (187h25m4.549387319s) [upload=164140]: Got next segment. Inline: false Dec 13 10:54:04.678334932 (187h25m4.549417967s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 13 10:54:04.678339972 (187h25m4.549423017s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 13 10:54:06.552039235 (187h25m6.42312228s) [seg_pos={0 0}, upload=164140]: Begin upload segment... Dec 13 10:54:06.552085274 (187h25m6.423168309s) [seg_pos={0 0}, upload=164140]: Done begin upload segment. Dec 13 10:54:06.552111302 (187h25m6.423194338s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552155768 (187h25m6.423238813s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552846924 (187h25m6.423929959s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553257215 (187h25m6.42434026s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553392723 (187h25m6.424475758s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553588406 (187h25m6.424671451s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553840846 (187h25m6.424923891s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553974851 (187h25m6.425057897s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554228795 (187h25m6.42531184s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554359133 (187h25m6.425442178s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554712255 (187h25m6.4257953s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554907297 (187h25m6.425990342s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555042745 (187h25m6.42612579s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555365890 (187h25m6.426448925s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555459569 (187h25m6.426542614s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555696730 (187h25m6.426779765s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:08.379289923 (187h25m8.250372968s) [upload=164140]: Next returned nil segment Dec 13 10:54:08.379316804 (187h25m8.25039985s) [upload=164140]: Waiting for error group managing segments... Dec 13 10:54:08.687944366 (187h25m8.559027411s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.688191346 (187h25m8.559274391s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.695282860 (187h25m8.566365905s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.699745637 (187h25m8.570828682s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719582961 (187h25m8.590666006s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719768304 (187h25m8.590851349s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720334432 (187h25m8.591417477s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720634563 (187h25m8.591717609s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.724972483 (187h25m8.596055528s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.739884178 (187h25m8.610967223s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.876599508 (187h25m8.747682553s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.943217551 (187h25m8.814300586s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000359268 (187h25m8.871442323s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000386250 (187h25m8.871469295s) [seg_pos={0 0}, upload=164140]: Segment reached optimal threshold of 13 pieces. Dec 13 10:54:09.000658989 (187h25m8.871742094s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection Dec 13 10:54:09.000669228 (187h25m8.871752273s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.000925507 (187h25m8.872008542s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: read tcp 188.166.225.170:42076->145.79.227.48:4302: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.000933592 (187h25m8.872016637s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001235117 (187h25m8.872318172s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: read tcp 188.166.225.170:36732->145.79.227.50:4374: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.001262077 (187h25m8.872345142s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001292736 (187h25m8.872375791s) [seg_pos={0 0}, upload=164140]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 13 10:54:09.001392175 (187h25m8.87247522s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 10:54:09.001399449 (187h25m8.872482494s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 10:54:09.331221628 (187h25m9.202304674s) [upload=164140]: Done uploading segments... Dec 13 10:54:09.331245344 (187h25m9.202328389s) [upload=164140]: Upload finished. err: <nil> Duration: 7.61823784s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 7.810706905s trace: https://jaeger.dev.storj.io/trace/7a99583e03db57d FAIL (2025-12-13 11:19:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ) context deadline exceeded; piecestore: (Node ID: 12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, Piece ID: EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA) context deadline exceeded; piecestore: (Node ID: 12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, Piece ID: AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ) context deadline exceeded ======LOGS====== Dec 13 10:54:04.678153217 (187h25m4.549236252s) [upload=164140]: Uploading segments... Dec 13 10:54:04.678304274 (187h25m4.549387319s) [upload=164140]: Got next segment. Inline: false Dec 13 10:54:04.678334932 (187h25m4.549417967s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 13 10:54:04.678339972 (187h25m4.549423017s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 13 10:54:06.552039235 (187h25m6.42312228s) [seg_pos={0 0}, upload=164140]: Begin upload segment... Dec 13 10:54:06.552085274 (187h25m6.423168309s) [seg_pos={0 0}, upload=164140]: Done begin upload segment. Dec 13 10:54:06.552111302 (187h25m6.423194338s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552155768 (187h25m6.423238813s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552846924 (187h25m6.423929959s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553257215 (187h25m6.42434026s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553392723 (187h25m6.424475758s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553588406 (187h25m6.424671451s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553840846 (187h25m6.424923891s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553974851 (187h25m6.425057897s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554228795 (187h25m6.42531184s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554359133 (187h25m6.425442178s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554712255 (187h25m6.4257953s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554907297 (187h25m6.425990342s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555042745 (187h25m6.42612579s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555365890 (187h25m6.426448925s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555459569 (187h25m6.426542614s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555696730 (187h25m6.426779765s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:08.379289923 (187h25m8.250372968s) [upload=164140]: Next returned nil segment Dec 13 10:54:08.379316804 (187h25m8.25039985s) [upload=164140]: Waiting for error group managing segments... Dec 13 10:54:08.687944366 (187h25m8.559027411s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.688191346 (187h25m8.559274391s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.695282860 (187h25m8.566365905s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.699745637 (187h25m8.570828682s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719582961 (187h25m8.590666006s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719768304 (187h25m8.590851349s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720334432 (187h25m8.591417477s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720634563 (187h25m8.591717609s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.724972483 (187h25m8.596055528s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.739884178 (187h25m8.610967223s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.876599508 (187h25m8.747682553s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.943217551 (187h25m8.814300586s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000359268 (187h25m8.871442323s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000386250 (187h25m8.871469295s) [seg_pos={0 0}, upload=164140]: Segment reached optimal threshold of 13 pieces. Dec 13 10:54:09.000658989 (187h25m8.871742094s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection Dec 13 10:54:09.000669228 (187h25m8.871752273s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.000925507 (187h25m8.872008542s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: read tcp 188.166.225.170:42076->145.79.227.48:4302: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.000933592 (187h25m8.872016637s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001235117 (187h25m8.872318172s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: read tcp 188.166.225.170:36732->145.79.227.50:4374: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.001262077 (187h25m8.872345142s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001292736 (187h25m8.872375791s) [seg_pos={0 0}, upload=164140]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 13 10:54:09.001392175 (187h25m8.87247522s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 10:54:09.001399449 (187h25m8.872482494s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 10:54:09.331221628 (187h25m9.202304674s) [upload=164140]: Done uploading segments... Dec 13 10:54:09.331245344 (187h25m9.202328389s) [upload=164140]: Upload finished. err: <nil> Duration: 7.810738637s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................X.....................X....................] duration: 4.868389802s trace: https://jaeger.dev.storj.io/trace/35e59be92effb9d0 FAIL (2025-12-13 11:19:14 UTC) orbiter warning: context deadline exceeded Duration: 4.868451689s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................X.....................X...................] duration: 5.000505511s trace: https://jaeger.dev.storj.io/trace/35e59be92effb9d0 FAIL (2025-12-13 11:18:25 UTC) orbiter warning: context deadline exceeded Duration: 5.000526721s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.948390433s trace: https://jaeger.dev.storj.io/trace/7a99583e03db57d FAIL (2025-12-13 11:18:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ) context deadline exceeded; piecestore: (Node ID: 12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, Piece ID: EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA) context deadline exceeded; piecestore: (Node ID: 12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, Piece ID: AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ) context deadline exceeded ======LOGS====== Dec 13 10:54:04.678153217 (187h25m4.549236252s) [upload=164140]: Uploading segments... Dec 13 10:54:04.678304274 (187h25m4.549387319s) [upload=164140]: Got next segment. Inline: false Dec 13 10:54:04.678334932 (187h25m4.549417967s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 13 10:54:04.678339972 (187h25m4.549423017s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 13 10:54:06.552039235 (187h25m6.42312228s) [seg_pos={0 0}, upload=164140]: Begin upload segment... Dec 13 10:54:06.552085274 (187h25m6.423168309s) [seg_pos={0 0}, upload=164140]: Done begin upload segment. Dec 13 10:54:06.552111302 (187h25m6.423194338s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552155768 (187h25m6.423238813s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552846924 (187h25m6.423929959s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553257215 (187h25m6.42434026s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553392723 (187h25m6.424475758s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553588406 (187h25m6.424671451s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553840846 (187h25m6.424923891s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553974851 (187h25m6.425057897s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554228795 (187h25m6.42531184s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554359133 (187h25m6.425442178s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554712255 (187h25m6.4257953s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554907297 (187h25m6.425990342s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555042745 (187h25m6.42612579s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555365890 (187h25m6.426448925s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555459569 (187h25m6.426542614s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555696730 (187h25m6.426779765s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:08.379289923 (187h25m8.250372968s) [upload=164140]: Next returned nil segment Dec 13 10:54:08.379316804 (187h25m8.25039985s) [upload=164140]: Waiting for error group managing segments... Dec 13 10:54:08.687944366 (187h25m8.559027411s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.688191346 (187h25m8.559274391s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.695282860 (187h25m8.566365905s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.699745637 (187h25m8.570828682s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719582961 (187h25m8.590666006s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719768304 (187h25m8.590851349s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720334432 (187h25m8.591417477s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720634563 (187h25m8.591717609s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.724972483 (187h25m8.596055528s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.739884178 (187h25m8.610967223s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.876599508 (187h25m8.747682553s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.943217551 (187h25m8.814300586s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000359268 (187h25m8.871442323s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000386250 (187h25m8.871469295s) [seg_pos={0 0}, upload=164140]: Segment reached optimal threshold of 13 pieces. Dec 13 10:54:09.000658989 (187h25m8.871742094s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection Dec 13 10:54:09.000669228 (187h25m8.871752273s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.000925507 (187h25m8.872008542s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: read tcp 188.166.225.170:42076->145.79.227.48:4302: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.000933592 (187h25m8.872016637s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001235117 (187h25m8.872318172s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: read tcp 188.166.225.170:36732->145.79.227.50:4374: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.001262077 (187h25m8.872345142s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001292736 (187h25m8.872375791s) [seg_pos={0 0}, upload=164140]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 13 10:54:09.001392175 (187h25m8.87247522s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 10:54:09.001399449 (187h25m8.872482494s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 10:54:09.331221628 (187h25m9.202304674s) [upload=164140]: Done uploading segments... Dec 13 10:54:09.331245344 (187h25m9.202328389s) [upload=164140]: Upload finished. err: <nil> Duration: 6.94842055s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...................X.....................X..................] duration: 3.380910517s trace: https://jaeger.dev.storj.io/trace/35e59be92effb9d0 FAIL (2025-12-13 11:17:21 UTC) orbiter warning: context deadline exceeded Duration: 3.380934983s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....................X.....................X.................] duration: 3.652212753s trace: https://jaeger.dev.storj.io/trace/35e59be92effb9d0 FAIL (2025-12-13 11:16:57 UTC) orbiter warning: context deadline exceeded Duration: 3.652257169s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 6.867758595s trace: https://jaeger.dev.storj.io/trace/7a99583e03db57d FAIL (2025-12-13 11:16:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ) context deadline exceeded; piecestore: (Node ID: 12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, Piece ID: EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA) context deadline exceeded; piecestore: (Node ID: 12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, Piece ID: AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ) context deadline exceeded ======LOGS====== Dec 13 10:54:04.678153217 (187h25m4.549236252s) [upload=164140]: Uploading segments... Dec 13 10:54:04.678304274 (187h25m4.549387319s) [upload=164140]: Got next segment. Inline: false Dec 13 10:54:04.678334932 (187h25m4.549417967s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 13 10:54:04.678339972 (187h25m4.549423017s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 13 10:54:06.552039235 (187h25m6.42312228s) [seg_pos={0 0}, upload=164140]: Begin upload segment... Dec 13 10:54:06.552085274 (187h25m6.423168309s) [seg_pos={0 0}, upload=164140]: Done begin upload segment. Dec 13 10:54:06.552111302 (187h25m6.423194338s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552155768 (187h25m6.423238813s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552846924 (187h25m6.423929959s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553257215 (187h25m6.42434026s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553392723 (187h25m6.424475758s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553588406 (187h25m6.424671451s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553840846 (187h25m6.424923891s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553974851 (187h25m6.425057897s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554228795 (187h25m6.42531184s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554359133 (187h25m6.425442178s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554712255 (187h25m6.4257953s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554907297 (187h25m6.425990342s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555042745 (187h25m6.42612579s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555365890 (187h25m6.426448925s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555459569 (187h25m6.426542614s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555696730 (187h25m6.426779765s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:08.379289923 (187h25m8.250372968s) [upload=164140]: Next returned nil segment Dec 13 10:54:08.379316804 (187h25m8.25039985s) [upload=164140]: Waiting for error group managing segments... Dec 13 10:54:08.687944366 (187h25m8.559027411s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.688191346 (187h25m8.559274391s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.695282860 (187h25m8.566365905s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.699745637 (187h25m8.570828682s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719582961 (187h25m8.590666006s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719768304 (187h25m8.590851349s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720334432 (187h25m8.591417477s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720634563 (187h25m8.591717609s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.724972483 (187h25m8.596055528s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.739884178 (187h25m8.610967223s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.876599508 (187h25m8.747682553s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.943217551 (187h25m8.814300586s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000359268 (187h25m8.871442323s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000386250 (187h25m8.871469295s) [seg_pos={0 0}, upload=164140]: Segment reached optimal threshold of 13 pieces. Dec 13 10:54:09.000658989 (187h25m8.871742094s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection Dec 13 10:54:09.000669228 (187h25m8.871752273s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.000925507 (187h25m8.872008542s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: read tcp 188.166.225.170:42076->145.79.227.48:4302: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.000933592 (187h25m8.872016637s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001235117 (187h25m8.872318172s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: read tcp 188.166.225.170:36732->145.79.227.50:4374: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.001262077 (187h25m8.872345142s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001292736 (187h25m8.872375791s) [seg_pos={0 0}, upload=164140]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 13 10:54:09.001392175 (187h25m8.87247522s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 10:54:09.001399449 (187h25m8.872482494s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 10:54:09.331221628 (187h25m9.202304674s) [upload=164140]: Done uploading segments... Dec 13 10:54:09.331245344 (187h25m9.202328389s) [upload=164140]: Upload finished. err: <nil> Duration: 6.86778723s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....................X.....................X................] duration: 4.127131565s trace: https://jaeger.dev.storj.io/trace/35e59be92effb9d0 FAIL (2025-12-13 11:16:05 UTC) orbiter warning: context deadline exceeded Duration: 4.127158636s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.793035439s trace: https://jaeger.dev.storj.io/trace/7a99583e03db57d FAIL (2025-12-13 11:15:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ) context deadline exceeded; piecestore: (Node ID: 12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, Piece ID: EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA) context deadline exceeded; piecestore: (Node ID: 12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, Piece ID: AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ) context deadline exceeded ======LOGS====== Dec 13 10:54:04.678153217 (187h25m4.549236252s) [upload=164140]: Uploading segments... Dec 13 10:54:04.678304274 (187h25m4.549387319s) [upload=164140]: Got next segment. Inline: false Dec 13 10:54:04.678334932 (187h25m4.549417967s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 13 10:54:04.678339972 (187h25m4.549423017s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 13 10:54:06.552039235 (187h25m6.42312228s) [seg_pos={0 0}, upload=164140]: Begin upload segment... Dec 13 10:54:06.552085274 (187h25m6.423168309s) [seg_pos={0 0}, upload=164140]: Done begin upload segment. Dec 13 10:54:06.552111302 (187h25m6.423194338s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552155768 (187h25m6.423238813s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552846924 (187h25m6.423929959s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553257215 (187h25m6.42434026s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553392723 (187h25m6.424475758s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553588406 (187h25m6.424671451s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553840846 (187h25m6.424923891s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553974851 (187h25m6.425057897s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554228795 (187h25m6.42531184s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554359133 (187h25m6.425442178s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554712255 (187h25m6.4257953s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554907297 (187h25m6.425990342s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555042745 (187h25m6.42612579s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555365890 (187h25m6.426448925s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555459569 (187h25m6.426542614s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555696730 (187h25m6.426779765s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:08.379289923 (187h25m8.250372968s) [upload=164140]: Next returned nil segment Dec 13 10:54:08.379316804 (187h25m8.25039985s) [upload=164140]: Waiting for error group managing segments... Dec 13 10:54:08.687944366 (187h25m8.559027411s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.688191346 (187h25m8.559274391s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.695282860 (187h25m8.566365905s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.699745637 (187h25m8.570828682s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719582961 (187h25m8.590666006s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719768304 (187h25m8.590851349s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720334432 (187h25m8.591417477s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720634563 (187h25m8.591717609s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.724972483 (187h25m8.596055528s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.739884178 (187h25m8.610967223s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.876599508 (187h25m8.747682553s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.943217551 (187h25m8.814300586s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000359268 (187h25m8.871442323s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000386250 (187h25m8.871469295s) [seg_pos={0 0}, upload=164140]: Segment reached optimal threshold of 13 pieces. Dec 13 10:54:09.000658989 (187h25m8.871742094s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection Dec 13 10:54:09.000669228 (187h25m8.871752273s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.000925507 (187h25m8.872008542s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: read tcp 188.166.225.170:42076->145.79.227.48:4302: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.000933592 (187h25m8.872016637s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001235117 (187h25m8.872318172s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: read tcp 188.166.225.170:36732->145.79.227.50:4374: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.001262077 (187h25m8.872345142s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001292736 (187h25m8.872375791s) [seg_pos={0 0}, upload=164140]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 13 10:54:09.001392175 (187h25m8.87247522s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 10:54:09.001399449 (187h25m8.872482494s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 10:54:09.331221628 (187h25m9.202304674s) [upload=164140]: Done uploading segments... Dec 13 10:54:09.331245344 (187h25m9.202328389s) [upload=164140]: Upload finished. err: <nil> Duration: 6.793059164s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 8.05096422s trace: https://jaeger.dev.storj.io/trace/7a99583e03db57d FAIL (2025-12-13 11:15:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ) context deadline exceeded; piecestore: (Node ID: 12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, Piece ID: EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA) context deadline exceeded; piecestore: (Node ID: 12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, Piece ID: AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ) context deadline exceeded ======LOGS====== Dec 13 10:54:04.678153217 (187h25m4.549236252s) [upload=164140]: Uploading segments... Dec 13 10:54:04.678304274 (187h25m4.549387319s) [upload=164140]: Got next segment. Inline: false Dec 13 10:54:04.678334932 (187h25m4.549417967s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 13 10:54:04.678339972 (187h25m4.549423017s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 13 10:54:06.552039235 (187h25m6.42312228s) [seg_pos={0 0}, upload=164140]: Begin upload segment... Dec 13 10:54:06.552085274 (187h25m6.423168309s) [seg_pos={0 0}, upload=164140]: Done begin upload segment. Dec 13 10:54:06.552111302 (187h25m6.423194338s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552155768 (187h25m6.423238813s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552846924 (187h25m6.423929959s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553257215 (187h25m6.42434026s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553392723 (187h25m6.424475758s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553588406 (187h25m6.424671451s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553840846 (187h25m6.424923891s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553974851 (187h25m6.425057897s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554228795 (187h25m6.42531184s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554359133 (187h25m6.425442178s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554712255 (187h25m6.4257953s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554907297 (187h25m6.425990342s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555042745 (187h25m6.42612579s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555365890 (187h25m6.426448925s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555459569 (187h25m6.426542614s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555696730 (187h25m6.426779765s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:08.379289923 (187h25m8.250372968s) [upload=164140]: Next returned nil segment Dec 13 10:54:08.379316804 (187h25m8.25039985s) [upload=164140]: Waiting for error group managing segments... Dec 13 10:54:08.687944366 (187h25m8.559027411s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.688191346 (187h25m8.559274391s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.695282860 (187h25m8.566365905s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.699745637 (187h25m8.570828682s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719582961 (187h25m8.590666006s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719768304 (187h25m8.590851349s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720334432 (187h25m8.591417477s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720634563 (187h25m8.591717609s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.724972483 (187h25m8.596055528s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.739884178 (187h25m8.610967223s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.876599508 (187h25m8.747682553s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.943217551 (187h25m8.814300586s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000359268 (187h25m8.871442323s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000386250 (187h25m8.871469295s) [seg_pos={0 0}, upload=164140]: Segment reached optimal threshold of 13 pieces. Dec 13 10:54:09.000658989 (187h25m8.871742094s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection Dec 13 10:54:09.000669228 (187h25m8.871752273s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.000925507 (187h25m8.872008542s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: read tcp 188.166.225.170:42076->145.79.227.48:4302: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.000933592 (187h25m8.872016637s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001235117 (187h25m8.872318172s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: read tcp 188.166.225.170:36732->145.79.227.50:4374: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.001262077 (187h25m8.872345142s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001292736 (187h25m8.872375791s) [seg_pos={0 0}, upload=164140]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 13 10:54:09.001392175 (187h25m8.87247522s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 10:54:09.001399449 (187h25m8.872482494s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 10:54:09.331221628 (187h25m9.202304674s) [upload=164140]: Done uploading segments... Dec 13 10:54:09.331245344 (187h25m9.202328389s) [upload=164140]: Upload finished. err: <nil> Duration: 8.050992893s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 7.712161489s trace: https://jaeger.dev.storj.io/trace/7a99583e03db57d FAIL (2025-12-13 11:15:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ) context deadline exceeded; piecestore: (Node ID: 12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, Piece ID: EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA) context deadline exceeded; piecestore: (Node ID: 12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, Piece ID: AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ) context deadline exceeded ======LOGS====== Dec 13 10:54:04.678153217 (187h25m4.549236252s) [upload=164140]: Uploading segments... Dec 13 10:54:04.678304274 (187h25m4.549387319s) [upload=164140]: Got next segment. Inline: false Dec 13 10:54:04.678334932 (187h25m4.549417967s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 13 10:54:04.678339972 (187h25m4.549423017s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 13 10:54:06.552039235 (187h25m6.42312228s) [seg_pos={0 0}, upload=164140]: Begin upload segment... Dec 13 10:54:06.552085274 (187h25m6.423168309s) [seg_pos={0 0}, upload=164140]: Done begin upload segment. Dec 13 10:54:06.552111302 (187h25m6.423194338s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552155768 (187h25m6.423238813s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552846924 (187h25m6.423929959s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553257215 (187h25m6.42434026s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553392723 (187h25m6.424475758s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553588406 (187h25m6.424671451s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553840846 (187h25m6.424923891s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553974851 (187h25m6.425057897s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554228795 (187h25m6.42531184s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554359133 (187h25m6.425442178s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554712255 (187h25m6.4257953s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554907297 (187h25m6.425990342s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555042745 (187h25m6.42612579s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555365890 (187h25m6.426448925s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555459569 (187h25m6.426542614s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555696730 (187h25m6.426779765s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:08.379289923 (187h25m8.250372968s) [upload=164140]: Next returned nil segment Dec 13 10:54:08.379316804 (187h25m8.25039985s) [upload=164140]: Waiting for error group managing segments... Dec 13 10:54:08.687944366 (187h25m8.559027411s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.688191346 (187h25m8.559274391s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.695282860 (187h25m8.566365905s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.699745637 (187h25m8.570828682s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719582961 (187h25m8.590666006s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719768304 (187h25m8.590851349s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720334432 (187h25m8.591417477s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720634563 (187h25m8.591717609s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.724972483 (187h25m8.596055528s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.739884178 (187h25m8.610967223s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.876599508 (187h25m8.747682553s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.943217551 (187h25m8.814300586s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000359268 (187h25m8.871442323s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000386250 (187h25m8.871469295s) [seg_pos={0 0}, upload=164140]: Segment reached optimal threshold of 13 pieces. Dec 13 10:54:09.000658989 (187h25m8.871742094s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection Dec 13 10:54:09.000669228 (187h25m8.871752273s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.000925507 (187h25m8.872008542s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: read tcp 188.166.225.170:42076->145.79.227.48:4302: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.000933592 (187h25m8.872016637s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001235117 (187h25m8.872318172s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: read tcp 188.166.225.170:36732->145.79.227.50:4374: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.001262077 (187h25m8.872345142s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001292736 (187h25m8.872375791s) [seg_pos={0 0}, upload=164140]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 13 10:54:09.001392175 (187h25m8.87247522s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 10:54:09.001399449 (187h25m8.872482494s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 10:54:09.331221628 (187h25m9.202304674s) [upload=164140]: Done uploading segments... Dec 13 10:54:09.331245344 (187h25m9.202328389s) [upload=164140]: Upload finished. err: <nil> Duration: 7.712189653s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......................X.....................X...............] duration: 3.448723514s trace: https://jaeger.dev.storj.io/trace/35e59be92effb9d0 FAIL (2025-12-13 11:15:02 UTC) orbiter warning: context deadline exceeded Duration: 3.44875347s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......................X.....................X..............] duration: 4.145931303s trace: https://jaeger.dev.storj.io/trace/35e59be92effb9d0 FAIL (2025-12-13 11:14:45 UTC) orbiter warning: context deadline exceeded Duration: 4.145958344s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 7.438570674s trace: https://jaeger.dev.storj.io/trace/7a99583e03db57d FAIL (2025-12-13 11:14:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ) context deadline exceeded; piecestore: (Node ID: 12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, Piece ID: EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA) context deadline exceeded; piecestore: (Node ID: 12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, Piece ID: AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ) context deadline exceeded ======LOGS====== Dec 13 10:54:04.678153217 (187h25m4.549236252s) [upload=164140]: Uploading segments... Dec 13 10:54:04.678304274 (187h25m4.549387319s) [upload=164140]: Got next segment. Inline: false Dec 13 10:54:04.678334932 (187h25m4.549417967s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 13 10:54:04.678339972 (187h25m4.549423017s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 13 10:54:06.552039235 (187h25m6.42312228s) [seg_pos={0 0}, upload=164140]: Begin upload segment... Dec 13 10:54:06.552085274 (187h25m6.423168309s) [seg_pos={0 0}, upload=164140]: Done begin upload segment. Dec 13 10:54:06.552111302 (187h25m6.423194338s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552155768 (187h25m6.423238813s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552846924 (187h25m6.423929959s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553257215 (187h25m6.42434026s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553392723 (187h25m6.424475758s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553588406 (187h25m6.424671451s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553840846 (187h25m6.424923891s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553974851 (187h25m6.425057897s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554228795 (187h25m6.42531184s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554359133 (187h25m6.425442178s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554712255 (187h25m6.4257953s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554907297 (187h25m6.425990342s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555042745 (187h25m6.42612579s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555365890 (187h25m6.426448925s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555459569 (187h25m6.426542614s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555696730 (187h25m6.426779765s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:08.379289923 (187h25m8.250372968s) [upload=164140]: Next returned nil segment Dec 13 10:54:08.379316804 (187h25m8.25039985s) [upload=164140]: Waiting for error group managing segments... Dec 13 10:54:08.687944366 (187h25m8.559027411s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.688191346 (187h25m8.559274391s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.695282860 (187h25m8.566365905s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.699745637 (187h25m8.570828682s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719582961 (187h25m8.590666006s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719768304 (187h25m8.590851349s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720334432 (187h25m8.591417477s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720634563 (187h25m8.591717609s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.724972483 (187h25m8.596055528s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.739884178 (187h25m8.610967223s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.876599508 (187h25m8.747682553s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.943217551 (187h25m8.814300586s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000359268 (187h25m8.871442323s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000386250 (187h25m8.871469295s) [seg_pos={0 0}, upload=164140]: Segment reached optimal threshold of 13 pieces. Dec 13 10:54:09.000658989 (187h25m8.871742094s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection Dec 13 10:54:09.000669228 (187h25m8.871752273s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.000925507 (187h25m8.872008542s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: read tcp 188.166.225.170:42076->145.79.227.48:4302: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.000933592 (187h25m8.872016637s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001235117 (187h25m8.872318172s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: read tcp 188.166.225.170:36732->145.79.227.50:4374: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.001262077 (187h25m8.872345142s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001292736 (187h25m8.872375791s) [seg_pos={0 0}, upload=164140]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 13 10:54:09.001392175 (187h25m8.87247522s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 10:54:09.001399449 (187h25m8.872482494s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 10:54:09.331221628 (187h25m9.202304674s) [upload=164140]: Done uploading segments... Dec 13 10:54:09.331245344 (187h25m9.202328389s) [upload=164140]: Upload finished. err: <nil> Duration: 7.438596804s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........................X.....................X.............] duration: 3.538736996s trace: https://jaeger.dev.storj.io/trace/35e59be92effb9d0 FAIL (2025-12-13 11:13:46 UTC) orbiter warning: context deadline exceeded Duration: 3.538762825s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.922023217s trace: https://jaeger.dev.storj.io/trace/7a99583e03db57d FAIL (2025-12-13 11:13:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ) context deadline exceeded; piecestore: (Node ID: 12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, Piece ID: EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA) context deadline exceeded; piecestore: (Node ID: 12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, Piece ID: AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ) context deadline exceeded ======LOGS====== Dec 13 10:54:04.678153217 (187h25m4.549236252s) [upload=164140]: Uploading segments... Dec 13 10:54:04.678304274 (187h25m4.549387319s) [upload=164140]: Got next segment. Inline: false Dec 13 10:54:04.678334932 (187h25m4.549417967s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 13 10:54:04.678339972 (187h25m4.549423017s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 13 10:54:06.552039235 (187h25m6.42312228s) [seg_pos={0 0}, upload=164140]: Begin upload segment... Dec 13 10:54:06.552085274 (187h25m6.423168309s) [seg_pos={0 0}, upload=164140]: Done begin upload segment. Dec 13 10:54:06.552111302 (187h25m6.423194338s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552155768 (187h25m6.423238813s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552846924 (187h25m6.423929959s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553257215 (187h25m6.42434026s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553392723 (187h25m6.424475758s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553588406 (187h25m6.424671451s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553840846 (187h25m6.424923891s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553974851 (187h25m6.425057897s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554228795 (187h25m6.42531184s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554359133 (187h25m6.425442178s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554712255 (187h25m6.4257953s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554907297 (187h25m6.425990342s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555042745 (187h25m6.42612579s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555365890 (187h25m6.426448925s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555459569 (187h25m6.426542614s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555696730 (187h25m6.426779765s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:08.379289923 (187h25m8.250372968s) [upload=164140]: Next returned nil segment Dec 13 10:54:08.379316804 (187h25m8.25039985s) [upload=164140]: Waiting for error group managing segments... Dec 13 10:54:08.687944366 (187h25m8.559027411s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.688191346 (187h25m8.559274391s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.695282860 (187h25m8.566365905s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.699745637 (187h25m8.570828682s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719582961 (187h25m8.590666006s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719768304 (187h25m8.590851349s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720334432 (187h25m8.591417477s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720634563 (187h25m8.591717609s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.724972483 (187h25m8.596055528s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.739884178 (187h25m8.610967223s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.876599508 (187h25m8.747682553s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.943217551 (187h25m8.814300586s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000359268 (187h25m8.871442323s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000386250 (187h25m8.871469295s) [seg_pos={0 0}, upload=164140]: Segment reached optimal threshold of 13 pieces. Dec 13 10:54:09.000658989 (187h25m8.871742094s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection Dec 13 10:54:09.000669228 (187h25m8.871752273s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.000925507 (187h25m8.872008542s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: read tcp 188.166.225.170:42076->145.79.227.48:4302: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.000933592 (187h25m8.872016637s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001235117 (187h25m8.872318172s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: read tcp 188.166.225.170:36732->145.79.227.50:4374: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.001262077 (187h25m8.872345142s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001292736 (187h25m8.872375791s) [seg_pos={0 0}, upload=164140]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 13 10:54:09.001392175 (187h25m8.87247522s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 10:54:09.001399449 (187h25m8.872482494s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 10:54:09.331221628 (187h25m9.202304674s) [upload=164140]: Done uploading segments... Dec 13 10:54:09.331245344 (187h25m9.202328389s) [upload=164140]: Upload finished. err: <nil> Duration: 5.922049748s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 7.678041388s trace: https://jaeger.dev.storj.io/trace/7a99583e03db57d FAIL (2025-12-13 11:13:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ) context deadline exceeded; piecestore: (Node ID: 12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, Piece ID: EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA) context deadline exceeded; piecestore: (Node ID: 12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, Piece ID: AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ) context deadline exceeded ======LOGS====== Dec 13 10:54:04.678153217 (187h25m4.549236252s) [upload=164140]: Uploading segments... Dec 13 10:54:04.678304274 (187h25m4.549387319s) [upload=164140]: Got next segment. Inline: false Dec 13 10:54:04.678334932 (187h25m4.549417967s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 13 10:54:04.678339972 (187h25m4.549423017s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 13 10:54:06.552039235 (187h25m6.42312228s) [seg_pos={0 0}, upload=164140]: Begin upload segment... Dec 13 10:54:06.552085274 (187h25m6.423168309s) [seg_pos={0 0}, upload=164140]: Done begin upload segment. Dec 13 10:54:06.552111302 (187h25m6.423194338s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552155768 (187h25m6.423238813s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552846924 (187h25m6.423929959s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553257215 (187h25m6.42434026s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553392723 (187h25m6.424475758s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553588406 (187h25m6.424671451s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553840846 (187h25m6.424923891s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553974851 (187h25m6.425057897s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554228795 (187h25m6.42531184s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554359133 (187h25m6.425442178s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554712255 (187h25m6.4257953s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554907297 (187h25m6.425990342s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555042745 (187h25m6.42612579s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555365890 (187h25m6.426448925s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555459569 (187h25m6.426542614s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555696730 (187h25m6.426779765s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:08.379289923 (187h25m8.250372968s) [upload=164140]: Next returned nil segment Dec 13 10:54:08.379316804 (187h25m8.25039985s) [upload=164140]: Waiting for error group managing segments... Dec 13 10:54:08.687944366 (187h25m8.559027411s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.688191346 (187h25m8.559274391s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.695282860 (187h25m8.566365905s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.699745637 (187h25m8.570828682s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719582961 (187h25m8.590666006s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719768304 (187h25m8.590851349s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720334432 (187h25m8.591417477s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720634563 (187h25m8.591717609s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.724972483 (187h25m8.596055528s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.739884178 (187h25m8.610967223s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.876599508 (187h25m8.747682553s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.943217551 (187h25m8.814300586s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000359268 (187h25m8.871442323s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000386250 (187h25m8.871469295s) [seg_pos={0 0}, upload=164140]: Segment reached optimal threshold of 13 pieces. Dec 13 10:54:09.000658989 (187h25m8.871742094s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection Dec 13 10:54:09.000669228 (187h25m8.871752273s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.000925507 (187h25m8.872008542s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: read tcp 188.166.225.170:42076->145.79.227.48:4302: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.000933592 (187h25m8.872016637s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001235117 (187h25m8.872318172s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: read tcp 188.166.225.170:36732->145.79.227.50:4374: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.001262077 (187h25m8.872345142s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001292736 (187h25m8.872375791s) [seg_pos={0 0}, upload=164140]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 13 10:54:09.001392175 (187h25m8.87247522s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 10:54:09.001399449 (187h25m8.872482494s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 10:54:09.331221628 (187h25m9.202304674s) [upload=164140]: Done uploading segments... Dec 13 10:54:09.331245344 (187h25m9.202328389s) [upload=164140]: Upload finished. err: <nil> Duration: 7.678071425s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........................X.....................X............] duration: 3.574486529s trace: https://jaeger.dev.storj.io/trace/35e59be92effb9d0 FAIL (2025-12-13 11:13:00 UTC) orbiter warning: context deadline exceeded Duration: 3.574527226s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.673368602s trace: https://jaeger.dev.storj.io/trace/7a99583e03db57d FAIL (2025-12-13 11:12:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ) context deadline exceeded; piecestore: (Node ID: 12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, Piece ID: EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA) context deadline exceeded; piecestore: (Node ID: 12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, Piece ID: AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ) context deadline exceeded ======LOGS====== Dec 13 10:54:04.678153217 (187h25m4.549236252s) [upload=164140]: Uploading segments... Dec 13 10:54:04.678304274 (187h25m4.549387319s) [upload=164140]: Got next segment. Inline: false Dec 13 10:54:04.678334932 (187h25m4.549417967s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 13 10:54:04.678339972 (187h25m4.549423017s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 13 10:54:06.552039235 (187h25m6.42312228s) [seg_pos={0 0}, upload=164140]: Begin upload segment... Dec 13 10:54:06.552085274 (187h25m6.423168309s) [seg_pos={0 0}, upload=164140]: Done begin upload segment. Dec 13 10:54:06.552111302 (187h25m6.423194338s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552155768 (187h25m6.423238813s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552846924 (187h25m6.423929959s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553257215 (187h25m6.42434026s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553392723 (187h25m6.424475758s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553588406 (187h25m6.424671451s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553840846 (187h25m6.424923891s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553974851 (187h25m6.425057897s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554228795 (187h25m6.42531184s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554359133 (187h25m6.425442178s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554712255 (187h25m6.4257953s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554907297 (187h25m6.425990342s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555042745 (187h25m6.42612579s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555365890 (187h25m6.426448925s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555459569 (187h25m6.426542614s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555696730 (187h25m6.426779765s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:08.379289923 (187h25m8.250372968s) [upload=164140]: Next returned nil segment Dec 13 10:54:08.379316804 (187h25m8.25039985s) [upload=164140]: Waiting for error group managing segments... Dec 13 10:54:08.687944366 (187h25m8.559027411s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.688191346 (187h25m8.559274391s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.695282860 (187h25m8.566365905s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.699745637 (187h25m8.570828682s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719582961 (187h25m8.590666006s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719768304 (187h25m8.590851349s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720334432 (187h25m8.591417477s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720634563 (187h25m8.591717609s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.724972483 (187h25m8.596055528s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.739884178 (187h25m8.610967223s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.876599508 (187h25m8.747682553s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.943217551 (187h25m8.814300586s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000359268 (187h25m8.871442323s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000386250 (187h25m8.871469295s) [seg_pos={0 0}, upload=164140]: Segment reached optimal threshold of 13 pieces. Dec 13 10:54:09.000658989 (187h25m8.871742094s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection Dec 13 10:54:09.000669228 (187h25m8.871752273s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.000925507 (187h25m8.872008542s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: read tcp 188.166.225.170:42076->145.79.227.48:4302: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.000933592 (187h25m8.872016637s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001235117 (187h25m8.872318172s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: read tcp 188.166.225.170:36732->145.79.227.50:4374: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.001262077 (187h25m8.872345142s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001292736 (187h25m8.872375791s) [seg_pos={0 0}, upload=164140]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 13 10:54:09.001392175 (187h25m8.87247522s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 10:54:09.001399449 (187h25m8.872482494s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 10:54:09.331221628 (187h25m9.202304674s) [upload=164140]: Done uploading segments... Dec 13 10:54:09.331245344 (187h25m9.202328389s) [upload=164140]: Upload finished. err: <nil> Duration: 6.673396484s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................X.....................X...........] duration: 5.000920913s trace: https://jaeger.dev.storj.io/trace/35e59be92effb9d0 FAIL (2025-12-13 11:11:55 UTC) orbiter warning: context deadline exceeded Duration: 5.000939167s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 7.684203311s trace: https://jaeger.dev.storj.io/trace/7a99583e03db57d FAIL (2025-12-13 11:11:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ) context deadline exceeded; piecestore: (Node ID: 12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, Piece ID: EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA) context deadline exceeded; piecestore: (Node ID: 12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, Piece ID: AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ) context deadline exceeded ======LOGS====== Dec 13 10:54:04.678153217 (187h25m4.549236252s) [upload=164140]: Uploading segments... Dec 13 10:54:04.678304274 (187h25m4.549387319s) [upload=164140]: Got next segment. Inline: false Dec 13 10:54:04.678334932 (187h25m4.549417967s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 13 10:54:04.678339972 (187h25m4.549423017s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 13 10:54:06.552039235 (187h25m6.42312228s) [seg_pos={0 0}, upload=164140]: Begin upload segment... Dec 13 10:54:06.552085274 (187h25m6.423168309s) [seg_pos={0 0}, upload=164140]: Done begin upload segment. Dec 13 10:54:06.552111302 (187h25m6.423194338s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552155768 (187h25m6.423238813s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552846924 (187h25m6.423929959s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553257215 (187h25m6.42434026s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553392723 (187h25m6.424475758s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553588406 (187h25m6.424671451s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553840846 (187h25m6.424923891s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553974851 (187h25m6.425057897s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554228795 (187h25m6.42531184s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554359133 (187h25m6.425442178s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554712255 (187h25m6.4257953s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554907297 (187h25m6.425990342s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555042745 (187h25m6.42612579s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555365890 (187h25m6.426448925s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555459569 (187h25m6.426542614s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555696730 (187h25m6.426779765s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:08.379289923 (187h25m8.250372968s) [upload=164140]: Next returned nil segment Dec 13 10:54:08.379316804 (187h25m8.25039985s) [upload=164140]: Waiting for error group managing segments... Dec 13 10:54:08.687944366 (187h25m8.559027411s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.688191346 (187h25m8.559274391s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.695282860 (187h25m8.566365905s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.699745637 (187h25m8.570828682s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719582961 (187h25m8.590666006s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719768304 (187h25m8.590851349s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720334432 (187h25m8.591417477s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720634563 (187h25m8.591717609s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.724972483 (187h25m8.596055528s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.739884178 (187h25m8.610967223s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.876599508 (187h25m8.747682553s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.943217551 (187h25m8.814300586s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000359268 (187h25m8.871442323s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000386250 (187h25m8.871469295s) [seg_pos={0 0}, upload=164140]: Segment reached optimal threshold of 13 pieces. Dec 13 10:54:09.000658989 (187h25m8.871742094s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection Dec 13 10:54:09.000669228 (187h25m8.871752273s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.000925507 (187h25m8.872008542s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: read tcp 188.166.225.170:42076->145.79.227.48:4302: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.000933592 (187h25m8.872016637s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001235117 (187h25m8.872318172s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: read tcp 188.166.225.170:36732->145.79.227.50:4374: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.001262077 (187h25m8.872345142s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001292736 (187h25m8.872375791s) [seg_pos={0 0}, upload=164140]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 13 10:54:09.001392175 (187h25m8.87247522s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 10:54:09.001399449 (187h25m8.872482494s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 10:54:09.331221628 (187h25m9.202304674s) [upload=164140]: Done uploading segments... Dec 13 10:54:09.331245344 (187h25m9.202328389s) [upload=164140]: Upload finished. err: <nil> Duration: 7.684229723s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........................X.....................X..........] duration: 3.487352904s trace: https://jaeger.dev.storj.io/trace/35e59be92effb9d0 FAIL (2025-12-13 11:11:04 UTC) orbiter warning: context deadline exceeded Duration: 3.487388152s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 7.225754524s trace: https://jaeger.dev.storj.io/trace/7a99583e03db57d FAIL (2025-12-13 11:11:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ) context deadline exceeded; piecestore: (Node ID: 12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, Piece ID: EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA) context deadline exceeded; piecestore: (Node ID: 12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, Piece ID: AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ) context deadline exceeded ======LOGS====== Dec 13 10:54:04.678153217 (187h25m4.549236252s) [upload=164140]: Uploading segments... Dec 13 10:54:04.678304274 (187h25m4.549387319s) [upload=164140]: Got next segment. Inline: false Dec 13 10:54:04.678334932 (187h25m4.549417967s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 13 10:54:04.678339972 (187h25m4.549423017s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 13 10:54:06.552039235 (187h25m6.42312228s) [seg_pos={0 0}, upload=164140]: Begin upload segment... Dec 13 10:54:06.552085274 (187h25m6.423168309s) [seg_pos={0 0}, upload=164140]: Done begin upload segment. Dec 13 10:54:06.552111302 (187h25m6.423194338s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552155768 (187h25m6.423238813s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.552846924 (187h25m6.423929959s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553257215 (187h25m6.42434026s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553392723 (187h25m6.424475758s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553588406 (187h25m6.424671451s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553840846 (187h25m6.424923891s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.553974851 (187h25m6.425057897s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554228795 (187h25m6.42531184s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554359133 (187h25m6.425442178s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554712255 (187h25m6.4257953s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.554907297 (187h25m6.425990342s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555042745 (187h25m6.42612579s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555365890 (187h25m6.426448925s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555459569 (187h25m6.426542614s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:06.555696730 (187h25m6.426779765s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Uploading piece... Dec 13 10:54:08.379289923 (187h25m8.250372968s) [upload=164140]: Next returned nil segment Dec 13 10:54:08.379316804 (187h25m8.25039985s) [upload=164140]: Waiting for error group managing segments... Dec 13 10:54:08.687944366 (187h25m8.559027411s) [noise=true , address=145.79.227.51:4410 , piece_id=P52CCPVHKKQ2CTSOK4ZKPEW3AJM4NOGFETIERJC2VBDQ2ZAXIDYA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.688191346 (187h25m8.559274391s) [noise=true , address=145.79.227.36:4519 , piece_id=NDJ67RQOVVVIGWQLJXYNQ3AJMX7NMCNJP4ZAQKWMI3UM7NNG524A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.695282860 (187h25m8.566365905s) [noise=true , address=145.79.227.40:4051 , piece_id=74E2LB332K7UJWWBYJA5L3PONAJOHPQWEFMGJGCGG7TGBCA7TQFA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.699745637 (187h25m8.570828682s) [noise=true , address=145.79.227.32:4834 , piece_id=NWYBWZCZKNZIH2JRGADMP37LFZBUA7GMAYZE5RFBMAPIF4MI47DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719582961 (187h25m8.590666006s) [noise=true , address=145.79.227.38:4979 , piece_id=O6QFVMXZMSDHRABJPMQ4YJK5UOGUMKDWUPT5FGPGSGIXAAFXNAGQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.719768304 (187h25m8.590851349s) [noise=true , address=145.79.227.27:4691 , piece_id=EVSH2NATNBH4SDE7VXBRYEPY3YZJR2LDXIY7N3PT75N7Z7LALUNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720334432 (187h25m8.591417477s) [noise=true , address=145.79.227.24:4583 , piece_id=BGTDY7RJKLY6DPD6FKZ3BVI2AD44CPNDETOPLRQ2SUHOXECTHE4Q, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.720634563 (187h25m8.591717609s) [noise=true , address=145.79.227.22:4511 , piece_id=VQKFVN7WUMZJAL3NPOSKXLZ5WAAFWNWQYL4BRBSQIO7KSSW5HGZQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.724972483 (187h25m8.596055528s) [noise=true , address=145.79.227.12:4767 , piece_id=UNLCWGU6J5WSDP7CEY6H5CAHBGIIOCD3KVLDPIS3VQZLXG5TVSNA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.739884178 (187h25m8.610967223s) [noise=true , address=145.79.227.42:4123 , piece_id=KA3FWKLANQ5CJYWQN422HL2C2HF76P4YZVATY2O2AJ5HSRO5G3DQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.876599508 (187h25m8.747682553s) [noise=true , address=77.90.31.19:4842 , piece_id=2HDY75OWSGHHZWNDKAOSIMPVQVOHEF5C2ULR4AXZHNDYKSPGOLEQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:08.943217551 (187h25m8.814300586s) [noise=true , address=145.79.227.27:4690 , piece_id=QJHULQR76TGGCD3DEWIJUPJXW2MYICISBW6GYJ5VVXNZOSVDBDDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000359268 (187h25m8.871442323s) [noise=true , address=145.79.227.25:4617 , piece_id=AMKOUHMJWBNC6DT7C677OLPRUGQSGZWHLVPJQTWILT4CHX5QNUDQ, seg_pos={0 0}, upload=164140]: Done uploading piece. err: <nil> Dec 13 10:54:09.000386250 (187h25m8.871469295s) [seg_pos={0 0}, upload=164140]: Segment reached optimal threshold of 13 pieces. Dec 13 10:54:09.000658989 (187h25m8.871742094s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33888->145.79.227.49:4339: use of closed network connection Dec 13 10:54:09.000669228 (187h25m8.871752273s) [noise=true , address=145.79.227.49:4339 , piece_id=2MB63LP64D6O5626BMROJB3Q3476WIGQO7TPHMBQRZ2KJF6WC7WA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.000925507 (187h25m8.872008542s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: read tcp 188.166.225.170:42076->145.79.227.48:4302: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.000933592 (187h25m8.872016637s) [noise=true , address=145.79.227.48:4302 , piece_id=DPNBASHHGZGJXWQDPKYPQAO6GXQCFXPZJ4TRYSPYKIROTZUNSR2A, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001235117 (187h25m8.872318172s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: read tcp 188.166.225.170:36732->145.79.227.50:4374: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 13 10:54:09.001262077 (187h25m8.872345142s) [noise=true , address=145.79.227.50:4374 , piece_id=W5EDVTLYE5PIGBHEGKCRD3L4P3G25AZIFKG4KOXTNODFRYJSCJAA, seg_pos={0 0}, upload=164140]: Long tail cancellation due to optimal threshold Dec 13 10:54:09.001292736 (187h25m8.872375791s) [seg_pos={0 0}, upload=164140]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 13 10:54:09.001392175 (187h25m8.87247522s) [upload=164140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 13 10:54:09.001399449 (187h25m8.872482494s) [upload=164140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 13 10:54:09.331221628 (187h25m9.202304674s) [upload=164140]: Done uploading segments... Dec 13 10:54:09.331245344 (187h25m9.202328389s) [upload=164140]: Upload finished. err: <nil> Duration: 7.225782257s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............................X.....................X.........] duration: 3.518454145s trace: https://jaeger.dev.storj.io/trace/35e59be92effb9d0 FAIL (2025-12-13 11:10:48 UTC) orbiter warning: context deadline exceeded Duration: 3.518482919s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)