Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [....XXXXXX...XX.....X...XX.X.XXX.XXXXX..X.X.X.X...XXXX.XX.X.] duration: 3.389706888s trace: https://jaeger.dev.storj.io/trace/4b02d94968e16f85 FAIL (2025-12-26 13:41:38 UTC) orbiter warning: context deadline exceeded Duration: 3.389731454s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 16 out of 60 is failed (threshold=3) [..X..X...................XX.X........X.XXX.X.X..X.X.X.XX....] duration: 52.580564593s trace: https://jaeger.dev.storj.io/trace/175ba4bb6e004432 FAIL (2025-12-26 13:41:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, Piece ID: 2LMZ6SVPT2ERLXTTPQYOFQDONGPSTUKRZPBZC463ZGZDGDQNKUJA) context deadline exceeded; piecestore: (Node ID: 12d7KuNgPYL5Ca2MPKQQCueQNSsYQifHXMhQ87VEXUhrKpLjuks, Piece ID: COXA5OSTIAYWL3C5IQSYUXYNFT7WCMOJ5EYQU5A2OJZMTXN27EWQ) context deadline exceeded; piecestore: (Node ID: 1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, Piece ID: 4KEC7MSAKLWFQ5LG27CWZHZWC4WYUOBVPSLHKZVVDO2HB2V72NIQ) context deadline exceeded; piecestore: (Node ID: 12eaTnUhynF1XeuvMr68SjMyYKM9rLj5tShzpViHcMSjvMs9EBF, Piece ID: 4NSWDLVEHASULNLN4L2SD7YJEOK2HGGSABSZKULLX6NVRACVF6DQ) context deadline exceeded ======LOGS====== Dec 26 13:36:37.629349713 (502h7m37.500432748s) [noise=true , address=77.90.31.13:4653 , piece_id=4NSWDLVEHASULNLN4L2SD7YJEOK2HGGSABSZKULLX6NVRACVF6DQ, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.629469422 (502h7m37.500552457s) [noise=true , address=77.90.31.21:4893 , piece_id=JJKUTOFJLGEALQT2ULTA2SOGJWCQ7RHA7NTTUD7VYQTI5BJS4C4Q, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.629613726 (502h7m37.500696771s) [noise=true , address=145.79.227.44:4223 , piece_id=SGRQEBXPVI5AVILWJMU6ZC4TSQSFQA4YL33BMK6LI3HZ6WNKV2NA, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.629945679 (502h7m37.501028714s) [noise=true , address=145.79.227.34:4915 , piece_id=Y3UN2P5C4JCZVQDPKD6QY5CE7Y2XKAJEJRW75RPAJ5ZZZRO53NIQ, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.630092078 (502h7m37.501175113s) [noise=true , address=145.79.227.30:4780 , piece_id=YPOZFLDB65TFHBXQKXW5B4QT5L7EKINWRPBN2IE4L76DBOLTMSZQ, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.630363495 (502h7m37.50144653s) [noise=true , address=45.91.116.12:4253 , piece_id=VOOVGVFPIXWOTCNAZHMVQAEBNAVOGSYDI7GN5W3H3EVW3UJTQ3ZA, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.630557453 (502h7m37.501640488s) [noise=true , address=45.91.116.9:4294 , piece_id=COXA5OSTIAYWL3C5IQSYUXYNFT7WCMOJ5EYQU5A2OJZMTXN27EWQ, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.630701678 (502h7m37.501784713s) [noise=true , address=64.49.13.169:4716 , piece_id=4YKHMLTI2MIJCADDUM7CMLDPQLJO6BKMXQULZ6RF5HKQ22YW5WJA, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.631024874 (502h7m37.502107919s) [noise=true , address=77.90.31.11:4560 , piece_id=NNZJTUPDWTMVIBRS6Z4UOU6SAA3KGXVCXWROS4YZDIXCEBJLBADA, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.631253629 (502h7m37.502336674s) [noise=true , address=145.79.227.43:4182 , piece_id=QAXA4SBNRU5K2ROO4EK66DACDN26P6Q6VSLNNHPZW2DOSAQHEF2A, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.631552238 (502h7m37.502635273s) [noise=true , address=145.79.227.9:4371 , piece_id=OJMMIDNJ7QTZAGAQD3GKHC57WGTWP4IIVPSXKV7Q252IXT3KNR3Q, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.631811722 (502h7m37.502894767s) [noise=true , address=145.79.227.52:4449 , piece_id=T6MFRO2B27SZYZA35D7CRVIYUGI3PMWDWKGAKO6WRN2UOCKV4RXA, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.632111463 (502h7m37.503194498s) [noise=true , address=145.79.227.51:4434 , piece_id=KHMZXFUE63HDONY5LJUKDR6LHYTEGYBQ5BHWKDKU4X5H5Y64Q6OA, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.632395784 (502h7m37.503478819s) [noise=true , address=77.90.31.10:4532 , piece_id=4KEC7MSAKLWFQ5LG27CWZHZWC4WYUOBVPSLHKZVVDO2HB2V72NIQ, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.634117873 (502h7m37.505200908s) [noise=true , address=45.91.116.4:4963 , piece_id=YCD5GJFEZMXAGAPZBYNM7SUR3ZDKM3ASKHMS5NEMRDZEDDF6DTGA, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.634545508 (502h7m37.505628543s) [noise=true , address=45.91.116.5:4327 , piece_id=ULSWEIDH4LFO5WGT7JIPEL4F7JSQBLWJARZUL44OBNLUC6LOUGOQ, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:39.098109889 (502h7m38.969192935s) [upload=440669]: Next returned nil segment Dec 26 13:36:39.098126871 (502h7m38.969209916s) [upload=440669]: Waiting for error group managing segments... Dec 26 13:36:39.504013736 (502h7m39.375096771s) [noise=true , address=145.79.227.51:4434 , piece_id=KHMZXFUE63HDONY5LJUKDR6LHYTEGYBQ5BHWKDKU4X5H5Y64Q6OA, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:39.513204096 (502h7m39.384287131s) [noise=true , address=145.79.227.52:4449 , piece_id=T6MFRO2B27SZYZA35D7CRVIYUGI3PMWDWKGAKO6WRN2UOCKV4RXA, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:39.532002612 (502h7m39.403085647s) [noise=true , address=145.79.227.43:4182 , piece_id=QAXA4SBNRU5K2ROO4EK66DACDN26P6Q6VSLNNHPZW2DOSAQHEF2A, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:39.591185697 (502h7m39.462268742s) [noise=true , address=145.79.227.9:4371 , piece_id=OJMMIDNJ7QTZAGAQD3GKHC57WGTWP4IIVPSXKV7Q252IXT3KNR3Q, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:39.851571721 (502h7m39.722654766s) [noise=true , address=45.91.116.9:4294 , piece_id=COXA5OSTIAYWL3C5IQSYUXYNFT7WCMOJ5EYQU5A2OJZMTXN27EWQ, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:39.897477856 (502h7m39.768560891s) [noise=true , address=77.90.31.11:4560 , piece_id=NNZJTUPDWTMVIBRS6Z4UOU6SAA3KGXVCXWROS4YZDIXCEBJLBADA, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:40.093563859 (502h7m39.964646905s) [noise=true , address=77.90.31.19:4856 , piece_id=2LMZ6SVPT2ERLXTTPQYOFQDONGPSTUKRZPBZC463ZGZDGDQNKUJA, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:40.406410178 (502h7m40.277493223s) [noise=true , address=145.79.227.44:4223 , piece_id=SGRQEBXPVI5AVILWJMU6ZC4TSQSFQA4YL33BMK6LI3HZ6WNKV2NA, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:40.535693498 (502h7m40.406776543s) [noise=true , address=145.79.227.34:4915 , piece_id=Y3UN2P5C4JCZVQDPKD6QY5CE7Y2XKAJEJRW75RPAJ5ZZZRO53NIQ, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:40.568325205 (502h7m40.43940825s) [noise=true , address=145.79.227.30:4780 , piece_id=YPOZFLDB65TFHBXQKXW5B4QT5L7EKINWRPBN2IE4L76DBOLTMSZQ, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:40.683137940 (502h7m40.554220975s) [noise=true , address=45.91.116.5:4327 , piece_id=ULSWEIDH4LFO5WGT7JIPEL4F7JSQBLWJARZUL44OBNLUC6LOUGOQ, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:40.700510821 (502h7m40.571593856s) [noise=true , address=77.90.31.13:4653 , piece_id=4NSWDLVEHASULNLN4L2SD7YJEOK2HGGSABSZKULLX6NVRACVF6DQ, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:40.794940292 (502h7m40.666023327s) [noise=true , address=77.90.31.10:4532 , piece_id=4KEC7MSAKLWFQ5LG27CWZHZWC4WYUOBVPSLHKZVVDO2HB2V72NIQ, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:40.825821145 (502h7m40.69690419s) [noise=true , address=77.90.31.12:4595 , piece_id=SGG2LCPF3VDF5JAF5PC7RREY35SR5F5AIWOACTLWSQZXLZUMDZKA, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:40.825846833 (502h7m40.696929869s) [seg_pos={0 0}, upload=440669]: Segment reached optimal threshold of 14 pieces. Dec 26 13:36:40.826053557 (502h7m40.697136592s) [noise=true , address=77.90.31.21:4893 , piece_id=JJKUTOFJLGEALQT2ULTA2SOGJWCQ7RHA7NTTUD7VYQTI5BJS4C4Q, seg_pos={0 0}, upload=440669]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC): protocol: expected piece hash; context canceled; EOF Dec 26 13:36:40.826061944 (502h7m40.697144989s) [noise=true , address=77.90.31.21:4893 , piece_id=JJKUTOFJLGEALQT2ULTA2SOGJWCQ7RHA7NTTUD7VYQTI5BJS4C4Q, seg_pos={0 0}, upload=440669]: Long tail cancellation due to optimal threshold Dec 26 13:36:40.826104385 (502h7m40.69718743s) [noise=true , address=64.49.13.169:4716 , piece_id=4YKHMLTI2MIJCADDUM7CMLDPQLJO6BKMXQULZ6RF5HKQ22YW5WJA, seg_pos={0 0}, upload=440669]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:49264->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:49264->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49264->64.49.13.169:4716: use of closed network connection Dec 26 13:36:40.826120344 (502h7m40.69720338s) [noise=true , address=64.49.13.169:4716 , piece_id=4YKHMLTI2MIJCADDUM7CMLDPQLJO6BKMXQULZ6RF5HKQ22YW5WJA, seg_pos={0 0}, upload=440669]: Long tail cancellation due to optimal threshold Dec 26 13:36:40.826190167 (502h7m40.697273202s) [noise=true , address=45.91.116.4:4963 , piece_id=YCD5GJFEZMXAGAPZBYNM7SUR3ZDKM3ASKHMS5NEMRDZEDDF6DTGA, seg_pos={0 0}, upload=440669]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; context canceled; EOF Dec 26 13:36:40.826196339 (502h7m40.697279384s) [noise=true , address=45.91.116.4:4963 , piece_id=YCD5GJFEZMXAGAPZBYNM7SUR3ZDKM3ASKHMS5NEMRDZEDDF6DTGA, seg_pos={0 0}, upload=440669]: Long tail cancellation due to optimal threshold Dec 26 13:36:40.826273687 (502h7m40.697356722s) [noise=true , address=45.91.116.12:4253 , piece_id=VOOVGVFPIXWOTCNAZHMVQAEBNAVOGSYDI7GN5W3H3EVW3UJTQ3ZA, seg_pos={0 0}, upload=440669]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gDvXYhShpWvptTU8bivGGwYBTyMF3PJcRv6RJ66vm8aZewejn): protocol: expected piece hash; context canceled; EOF Dec 26 13:36:40.826279708 (502h7m40.697362743s) [noise=true , address=45.91.116.12:4253 , piece_id=VOOVGVFPIXWOTCNAZHMVQAEBNAVOGSYDI7GN5W3H3EVW3UJTQ3ZA, seg_pos={0 0}, upload=440669]: Long tail cancellation due to optimal threshold Dec 26 13:36:40.826308654 (502h7m40.697391689s) [seg_pos={0 0}, upload=440669]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 13:36:40.826430335 (502h7m40.69751337s) [upload=440669]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:36:40.826435234 (502h7m40.697518279s) [upload=440669]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:36:41.158299667 (502h7m41.029382702s) [upload=440669]: Done uploading segments... Dec 26 13:36:41.158324554 (502h7m41.0294076s) [upload=440669]: Upload finished. err: <nil> Duration: 52.58058342s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [XXXXXX..X....X..X.X....X......XX.X.X......X.X.XX.X.X..X.XXXX] duration: 30.003800976s trace: https://jaeger.dev.storj.io/trace/638744980c71ef22 FAIL (2025-12-26 13:41:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: H2QOLP6UGKJ63NSDUO2OO75OQVCPFH53VOR4BGVVEBBF63KD7SRA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: G3MZQNR4IOGUAPRXT2YXTK3BND53KEOM5QLJO75QOZX3Q5CYFVFQ) context deadline exceeded ======LOGS====== Dec 26 13:40:54.585471478 (502h11m54.456554523s) [upload=440726]: Starting upload Dec 26 13:40:54.585531923 (502h11m54.456614958s) [upload=440726]: Done starting upload Dec 26 13:40:54.585831394 (502h11m54.456914429s) [upload=440726]: Uploading segments... Dec 26 13:40:54.585968264 (502h11m54.457051309s) [upload=440726]: Got next segment. Inline: false Dec 26 13:40:54.586017979 (502h11m54.457101014s) [upload=440726]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 13:40:54.586025974 (502h11m54.457109009s) [upload=440726]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 13:40:54.831308979 (502h11m54.702392014s) [seg_pos={0 0}, upload=440726]: Begin upload segment... Dec 26 13:40:54.831382590 (502h11m54.702465635s) [seg_pos={0 0}, upload=440726]: Done begin upload segment. Dec 26 13:40:54.831428637 (502h11m54.702511682s) [noise=true , address=145.79.227.51:4411 , piece_id=VANX4GQ3NLZKBQ3EQZSR2O2J5XNNCTBZCERWVQY3RACL55SVUOGQ, seg_pos={0 0}, upload=440726]: Uploading piece... Dec 26 13:40:54.831437975 (502h11m54.70252101s) [noise=true , address=145.79.227.31:4799 , piece_id=CTEYD6FZ3D3MVCICXCDXBDJPIWVBGPZEOURJFUNSZ3NQ23XBLU7A, seg_pos={0 0}, upload=440726]: Uploading piece... Dec 26 13:40:54.831968755 (502h11m54.7030518s) [noise=true , address=45.91.116.5:4300 , piece_id=G3MZQNR4IOGUAPRXT2YXTK3BND53KEOM5QLJO75QOZX3Q5CYFVFQ, seg_pos={0 0}, upload=440726]: Uploading piece... Dec 26 13:40:54.832385058 (502h11m54.703468103s) [noise=true , address=145.79.227.35:4483 , piece_id=QFFGIFO4L5YSQ6UVMP7N4L3XUQ2NHOCZO3XK75UC5RHDWFCC7SHQ, seg_pos={0 0}, upload=440726]: Uploading piece... Dec 26 13:40:54.832786092 (502h11m54.703869127s) [noise=true , address=145.79.227.19:4078 , piece_id=ABK45654SCO3OHQQOU6UCSOOBVSAEFQQ4IMU5JYR4ECFIA4WR72A, seg_pos={0 0}, upload=440726]: Uploading piece... Dec 26 13:40:54.833127021 (502h11m54.704210056s) [noise=true , address=145.79.227.45:4231 , piece_id=EV6D76A5MBHFGV7F63QCK5ZAISCLFFXN5FIYCV3A4DNNJRPEJXZA, seg_pos={0 0}, upload=440726]: Uploading piece... Dec 26 13:40:54.833430779 (502h11m54.704513814s) [noise=true , address=145.79.227.38:4978 , piece_id=ZDHIP3NOBTZ7W575QA7M6IPVH27V2W2TQFVCINLBVKCFP33OJIIA, seg_pos={0 0}, upload=440726]: Uploading piece... Dec 26 13:40:54.835374611 (502h11m54.706457646s) [noise=true , address=77.90.31.14:4661 , piece_id=FNVD2EB6XGTVDHKC472RJWEQZE3K2DQK3VRIOKHVG6TZQEOISKPA, seg_pos={0 0}, upload=440726]: Uploading piece... Dec 26 13:40:54.836472742 (502h11m54.707555777s) [noise=true , address=145.79.227.4:4185 , piece_id=JJ6JCJMCC5HCMIQJFBGK75D6DTNLWLJ3WDM4QB5EQ337CJ72THJA, seg_pos={0 0}, upload=440726]: Uploading piece... Dec 26 13:40:54.837557588 (502h11m54.708640623s) [noise=true , address=145.79.227.20:4438 , piece_id=2FA2XQDNUHN57OE5E4RRZGIKBJ56CPJUADER4SXDDZJTWSHJCSMQ, seg_pos={0 0}, upload=440726]: Uploading piece... Dec 26 13:40:54.838216773 (502h11m54.709299818s) [noise=true , address=145.79.227.48:4303 , piece_id=4ZI26RDFH7UTXXPLW55YS7VNHFTOLWFZMOKS5II2C7YQUJRRHANA, seg_pos={0 0}, upload=440726]: Uploading piece... Dec 26 13:40:54.840187907 (502h11m54.711270942s) [noise=true , address=145.79.227.44:4195 , piece_id=VVR2347Q5QMJWS5QBKFXUTJUCNGSNZ4ZN6OJYS3KCDF2U5ZBY4LA, seg_pos={0 0}, upload=440726]: Uploading piece... Dec 26 13:40:54.841357633 (502h11m54.712440668s) [noise=true , address=45.91.116.14:4190 , piece_id=BN5BTFEDFJPMFR42TULBIDXBZ6UV2ITMFZKTHETY23HUBWZCVQSA, seg_pos={0 0}, upload=440726]: Uploading piece... Dec 26 13:40:54.842550314 (502h11m54.713633349s) [noise=true , address=145.79.227.6:4258 , piece_id=BV5MURWJTDY2IWHKPTJCUH7DB5T5ROTN64ABFGDEL6PF7KXMIUHQ, seg_pos={0 0}, upload=440726]: Uploading piece... Dec 26 13:40:54.843730231 (502h11m54.714813276s) [noise=true , address=45.91.116.13:4154 , piece_id=H2QOLP6UGKJ63NSDUO2OO75OQVCPFH53VOR4BGVVEBBF63KD7SRA, seg_pos={0 0}, upload=440726]: Uploading piece... Dec 26 13:40:54.843855119 (502h11m54.714938164s) [noise=true , address=145.79.227.47:4265 , piece_id=6CUJ2TAS7WVX3ZSI44XDRYEPCKPTVDWVZIDEDX3JPMI3OAFPOLQA, seg_pos={0 0}, upload=440726]: Uploading piece... Dec 26 13:40:56.552702266 (502h11m56.423785301s) [upload=440726]: Next returned nil segment Dec 26 13:40:56.552715091 (502h11m56.423798126s) [upload=440726]: Waiting for error group managing segments... Dec 26 13:40:56.840657597 (502h11m56.711740632s) [noise=true , address=145.79.227.31:4799 , piece_id=CTEYD6FZ3D3MVCICXCDXBDJPIWVBGPZEOURJFUNSZ3NQ23XBLU7A, seg_pos={0 0}, upload=440726]: Done uploading piece. err: <nil> Dec 26 13:40:56.843868731 (502h11m56.714951777s) [noise=true , address=145.79.227.35:4483 , piece_id=QFFGIFO4L5YSQ6UVMP7N4L3XUQ2NHOCZO3XK75UC5RHDWFCC7SHQ, seg_pos={0 0}, upload=440726]: Done uploading piece. err: <nil> Dec 26 13:40:56.843971838 (502h11m56.715054883s) [noise=true , address=145.79.227.6:4258 , piece_id=BV5MURWJTDY2IWHKPTJCUH7DB5T5ROTN64ABFGDEL6PF7KXMIUHQ, seg_pos={0 0}, upload=440726]: Done uploading piece. err: <nil> Dec 26 13:40:56.847585529 (502h11m56.718668574s) [noise=true , address=145.79.227.45:4231 , piece_id=EV6D76A5MBHFGV7F63QCK5ZAISCLFFXN5FIYCV3A4DNNJRPEJXZA, seg_pos={0 0}, upload=440726]: Done uploading piece. err: <nil> Dec 26 13:40:56.851021692 (502h11m56.722104727s) [noise=true , address=145.79.227.48:4303 , piece_id=4ZI26RDFH7UTXXPLW55YS7VNHFTOLWFZMOKS5II2C7YQUJRRHANA, seg_pos={0 0}, upload=440726]: Done uploading piece. err: <nil> Dec 26 13:40:56.852998135 (502h11m56.72408118s) [noise=true , address=145.79.227.51:4411 , piece_id=VANX4GQ3NLZKBQ3EQZSR2O2J5XNNCTBZCERWVQY3RACL55SVUOGQ, seg_pos={0 0}, upload=440726]: Done uploading piece. err: <nil> Dec 26 13:40:56.857371754 (502h11m56.728454799s) [noise=true , address=145.79.227.19:4078 , piece_id=ABK45654SCO3OHQQOU6UCSOOBVSAEFQQ4IMU5JYR4ECFIA4WR72A, seg_pos={0 0}, upload=440726]: Done uploading piece. err: <nil> Dec 26 13:40:56.858032883 (502h11m56.729115918s) [noise=true , address=145.79.227.4:4185 , piece_id=JJ6JCJMCC5HCMIQJFBGK75D6DTNLWLJ3WDM4QB5EQ337CJ72THJA, seg_pos={0 0}, upload=440726]: Done uploading piece. err: <nil> Dec 26 13:40:56.874498175 (502h11m56.74558121s) [noise=true , address=145.79.227.20:4438 , piece_id=2FA2XQDNUHN57OE5E4RRZGIKBJ56CPJUADER4SXDDZJTWSHJCSMQ, seg_pos={0 0}, upload=440726]: Done uploading piece. err: <nil> Dec 26 13:40:56.876363077 (502h11m56.747446112s) [noise=true , address=145.79.227.38:4978 , piece_id=ZDHIP3NOBTZ7W575QA7M6IPVH27V2W2TQFVCINLBVKCFP33OJIIA, seg_pos={0 0}, upload=440726]: Done uploading piece. err: <nil> Dec 26 13:40:56.877960719 (502h11m56.749043764s) [noise=true , address=145.79.227.47:4265 , piece_id=6CUJ2TAS7WVX3ZSI44XDRYEPCKPTVDWVZIDEDX3JPMI3OAFPOLQA, seg_pos={0 0}, upload=440726]: Done uploading piece. err: <nil> Dec 26 13:40:57.104272860 (502h11m56.975355905s) [noise=true , address=45.91.116.13:4154 , piece_id=H2QOLP6UGKJ63NSDUO2OO75OQVCPFH53VOR4BGVVEBBF63KD7SRA, seg_pos={0 0}, upload=440726]: Done uploading piece. err: <nil> Dec 26 13:40:57.183006633 (502h11m57.054089668s) [noise=true , address=45.91.116.5:4300 , piece_id=G3MZQNR4IOGUAPRXT2YXTK3BND53KEOM5QLJO75QOZX3Q5CYFVFQ, seg_pos={0 0}, upload=440726]: Done uploading piece. err: <nil> Dec 26 13:40:57.183027493 (502h11m57.054110528s) [seg_pos={0 0}, upload=440726]: Segment reached optimal threshold of 13 pieces. Dec 26 13:40:57.183265245 (502h11m57.05434829s) [noise=true , address=77.90.31.14:4661 , piece_id=FNVD2EB6XGTVDHKC472RJWEQZE3K2DQK3VRIOKHVG6TZQEOISKPA, seg_pos={0 0}, upload=440726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 26 13:40:57.183277139 (502h11m57.054360184s) [noise=true , address=77.90.31.14:4661 , piece_id=FNVD2EB6XGTVDHKC472RJWEQZE3K2DQK3VRIOKHVG6TZQEOISKPA, seg_pos={0 0}, upload=440726]: Long tail cancellation due to optimal threshold Dec 26 13:40:57.183570867 (502h11m57.054653902s) [noise=true , address=45.91.116.14:4190 , piece_id=BN5BTFEDFJPMFR42TULBIDXBZ6UV2ITMFZKTHETY23HUBWZCVQSA, seg_pos={0 0}, upload=440726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 26 13:40:57.183580126 (502h11m57.054663171s) [noise=true , address=45.91.116.14:4190 , piece_id=BN5BTFEDFJPMFR42TULBIDXBZ6UV2ITMFZKTHETY23HUBWZCVQSA, seg_pos={0 0}, upload=440726]: Long tail cancellation due to optimal threshold Dec 26 13:40:57.183605133 (502h11m57.054688208s) [noise=true , address=145.79.227.44:4195 , piece_id=VVR2347Q5QMJWS5QBKFXUTJUCNGSNZ4ZN6OJYS3KCDF2U5ZBY4LA, seg_pos={0 0}, upload=440726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 26 13:40:57.183625111 (502h11m57.054708146s) [noise=true , address=145.79.227.44:4195 , piece_id=VVR2347Q5QMJWS5QBKFXUTJUCNGSNZ4ZN6OJYS3KCDF2U5ZBY4LA, seg_pos={0 0}, upload=440726]: Long tail cancellation due to optimal threshold Dec 26 13:40:57.183656060 (502h11m57.054739105s) [seg_pos={0 0}, upload=440726]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 13:40:57.183742194 (502h11m57.054825229s) [upload=440726]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:40:57.183749848 (502h11m57.054832883s) [upload=440726]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:40:57.486113145 (502h11m57.35719621s) [upload=440726]: Done uploading segments... Dec 26 13:40:57.486135347 (502h11m57.357218392s) [upload=440726]: Upload finished. err: <nil> Duration: 30.003829079s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 16 out of 60 is failed (threshold=3) [...X..X...................XX.X........X.XXX.X.X..X.X.X.XX...] duration: 56.114821353s trace: https://jaeger.dev.storj.io/trace/175ba4bb6e004432 FAIL (2025-12-26 13:40:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, Piece ID: 2LMZ6SVPT2ERLXTTPQYOFQDONGPSTUKRZPBZC463ZGZDGDQNKUJA) context deadline exceeded; piecestore: (Node ID: 12d7KuNgPYL5Ca2MPKQQCueQNSsYQifHXMhQ87VEXUhrKpLjuks, Piece ID: COXA5OSTIAYWL3C5IQSYUXYNFT7WCMOJ5EYQU5A2OJZMTXN27EWQ) context deadline exceeded; piecestore: (Node ID: 1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, Piece ID: 4KEC7MSAKLWFQ5LG27CWZHZWC4WYUOBVPSLHKZVVDO2HB2V72NIQ) context deadline exceeded; piecestore: (Node ID: 12eaTnUhynF1XeuvMr68SjMyYKM9rLj5tShzpViHcMSjvMs9EBF, Piece ID: 4NSWDLVEHASULNLN4L2SD7YJEOK2HGGSABSZKULLX6NVRACVF6DQ) context deadline exceeded ======LOGS====== Dec 26 13:36:37.629349713 (502h7m37.500432748s) [noise=true , address=77.90.31.13:4653 , piece_id=4NSWDLVEHASULNLN4L2SD7YJEOK2HGGSABSZKULLX6NVRACVF6DQ, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.629469422 (502h7m37.500552457s) [noise=true , address=77.90.31.21:4893 , piece_id=JJKUTOFJLGEALQT2ULTA2SOGJWCQ7RHA7NTTUD7VYQTI5BJS4C4Q, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.629613726 (502h7m37.500696771s) [noise=true , address=145.79.227.44:4223 , piece_id=SGRQEBXPVI5AVILWJMU6ZC4TSQSFQA4YL33BMK6LI3HZ6WNKV2NA, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.629945679 (502h7m37.501028714s) [noise=true , address=145.79.227.34:4915 , piece_id=Y3UN2P5C4JCZVQDPKD6QY5CE7Y2XKAJEJRW75RPAJ5ZZZRO53NIQ, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.630092078 (502h7m37.501175113s) [noise=true , address=145.79.227.30:4780 , piece_id=YPOZFLDB65TFHBXQKXW5B4QT5L7EKINWRPBN2IE4L76DBOLTMSZQ, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.630363495 (502h7m37.50144653s) [noise=true , address=45.91.116.12:4253 , piece_id=VOOVGVFPIXWOTCNAZHMVQAEBNAVOGSYDI7GN5W3H3EVW3UJTQ3ZA, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.630557453 (502h7m37.501640488s) [noise=true , address=45.91.116.9:4294 , piece_id=COXA5OSTIAYWL3C5IQSYUXYNFT7WCMOJ5EYQU5A2OJZMTXN27EWQ, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.630701678 (502h7m37.501784713s) [noise=true , address=64.49.13.169:4716 , piece_id=4YKHMLTI2MIJCADDUM7CMLDPQLJO6BKMXQULZ6RF5HKQ22YW5WJA, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.631024874 (502h7m37.502107919s) [noise=true , address=77.90.31.11:4560 , piece_id=NNZJTUPDWTMVIBRS6Z4UOU6SAA3KGXVCXWROS4YZDIXCEBJLBADA, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.631253629 (502h7m37.502336674s) [noise=true , address=145.79.227.43:4182 , piece_id=QAXA4SBNRU5K2ROO4EK66DACDN26P6Q6VSLNNHPZW2DOSAQHEF2A, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.631552238 (502h7m37.502635273s) [noise=true , address=145.79.227.9:4371 , piece_id=OJMMIDNJ7QTZAGAQD3GKHC57WGTWP4IIVPSXKV7Q252IXT3KNR3Q, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.631811722 (502h7m37.502894767s) [noise=true , address=145.79.227.52:4449 , piece_id=T6MFRO2B27SZYZA35D7CRVIYUGI3PMWDWKGAKO6WRN2UOCKV4RXA, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.632111463 (502h7m37.503194498s) [noise=true , address=145.79.227.51:4434 , piece_id=KHMZXFUE63HDONY5LJUKDR6LHYTEGYBQ5BHWKDKU4X5H5Y64Q6OA, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.632395784 (502h7m37.503478819s) [noise=true , address=77.90.31.10:4532 , piece_id=4KEC7MSAKLWFQ5LG27CWZHZWC4WYUOBVPSLHKZVVDO2HB2V72NIQ, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.634117873 (502h7m37.505200908s) [noise=true , address=45.91.116.4:4963 , piece_id=YCD5GJFEZMXAGAPZBYNM7SUR3ZDKM3ASKHMS5NEMRDZEDDF6DTGA, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.634545508 (502h7m37.505628543s) [noise=true , address=45.91.116.5:4327 , piece_id=ULSWEIDH4LFO5WGT7JIPEL4F7JSQBLWJARZUL44OBNLUC6LOUGOQ, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:39.098109889 (502h7m38.969192935s) [upload=440669]: Next returned nil segment Dec 26 13:36:39.098126871 (502h7m38.969209916s) [upload=440669]: Waiting for error group managing segments... Dec 26 13:36:39.504013736 (502h7m39.375096771s) [noise=true , address=145.79.227.51:4434 , piece_id=KHMZXFUE63HDONY5LJUKDR6LHYTEGYBQ5BHWKDKU4X5H5Y64Q6OA, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:39.513204096 (502h7m39.384287131s) [noise=true , address=145.79.227.52:4449 , piece_id=T6MFRO2B27SZYZA35D7CRVIYUGI3PMWDWKGAKO6WRN2UOCKV4RXA, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:39.532002612 (502h7m39.403085647s) [noise=true , address=145.79.227.43:4182 , piece_id=QAXA4SBNRU5K2ROO4EK66DACDN26P6Q6VSLNNHPZW2DOSAQHEF2A, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:39.591185697 (502h7m39.462268742s) [noise=true , address=145.79.227.9:4371 , piece_id=OJMMIDNJ7QTZAGAQD3GKHC57WGTWP4IIVPSXKV7Q252IXT3KNR3Q, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:39.851571721 (502h7m39.722654766s) [noise=true , address=45.91.116.9:4294 , piece_id=COXA5OSTIAYWL3C5IQSYUXYNFT7WCMOJ5EYQU5A2OJZMTXN27EWQ, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:39.897477856 (502h7m39.768560891s) [noise=true , address=77.90.31.11:4560 , piece_id=NNZJTUPDWTMVIBRS6Z4UOU6SAA3KGXVCXWROS4YZDIXCEBJLBADA, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:40.093563859 (502h7m39.964646905s) [noise=true , address=77.90.31.19:4856 , piece_id=2LMZ6SVPT2ERLXTTPQYOFQDONGPSTUKRZPBZC463ZGZDGDQNKUJA, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:40.406410178 (502h7m40.277493223s) [noise=true , address=145.79.227.44:4223 , piece_id=SGRQEBXPVI5AVILWJMU6ZC4TSQSFQA4YL33BMK6LI3HZ6WNKV2NA, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:40.535693498 (502h7m40.406776543s) [noise=true , address=145.79.227.34:4915 , piece_id=Y3UN2P5C4JCZVQDPKD6QY5CE7Y2XKAJEJRW75RPAJ5ZZZRO53NIQ, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:40.568325205 (502h7m40.43940825s) [noise=true , address=145.79.227.30:4780 , piece_id=YPOZFLDB65TFHBXQKXW5B4QT5L7EKINWRPBN2IE4L76DBOLTMSZQ, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:40.683137940 (502h7m40.554220975s) [noise=true , address=45.91.116.5:4327 , piece_id=ULSWEIDH4LFO5WGT7JIPEL4F7JSQBLWJARZUL44OBNLUC6LOUGOQ, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:40.700510821 (502h7m40.571593856s) [noise=true , address=77.90.31.13:4653 , piece_id=4NSWDLVEHASULNLN4L2SD7YJEOK2HGGSABSZKULLX6NVRACVF6DQ, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:40.794940292 (502h7m40.666023327s) [noise=true , address=77.90.31.10:4532 , piece_id=4KEC7MSAKLWFQ5LG27CWZHZWC4WYUOBVPSLHKZVVDO2HB2V72NIQ, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:40.825821145 (502h7m40.69690419s) [noise=true , address=77.90.31.12:4595 , piece_id=SGG2LCPF3VDF5JAF5PC7RREY35SR5F5AIWOACTLWSQZXLZUMDZKA, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:40.825846833 (502h7m40.696929869s) [seg_pos={0 0}, upload=440669]: Segment reached optimal threshold of 14 pieces. Dec 26 13:36:40.826053557 (502h7m40.697136592s) [noise=true , address=77.90.31.21:4893 , piece_id=JJKUTOFJLGEALQT2ULTA2SOGJWCQ7RHA7NTTUD7VYQTI5BJS4C4Q, seg_pos={0 0}, upload=440669]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC): protocol: expected piece hash; context canceled; EOF Dec 26 13:36:40.826061944 (502h7m40.697144989s) [noise=true , address=77.90.31.21:4893 , piece_id=JJKUTOFJLGEALQT2ULTA2SOGJWCQ7RHA7NTTUD7VYQTI5BJS4C4Q, seg_pos={0 0}, upload=440669]: Long tail cancellation due to optimal threshold Dec 26 13:36:40.826104385 (502h7m40.69718743s) [noise=true , address=64.49.13.169:4716 , piece_id=4YKHMLTI2MIJCADDUM7CMLDPQLJO6BKMXQULZ6RF5HKQ22YW5WJA, seg_pos={0 0}, upload=440669]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:49264->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:49264->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49264->64.49.13.169:4716: use of closed network connection Dec 26 13:36:40.826120344 (502h7m40.69720338s) [noise=true , address=64.49.13.169:4716 , piece_id=4YKHMLTI2MIJCADDUM7CMLDPQLJO6BKMXQULZ6RF5HKQ22YW5WJA, seg_pos={0 0}, upload=440669]: Long tail cancellation due to optimal threshold Dec 26 13:36:40.826190167 (502h7m40.697273202s) [noise=true , address=45.91.116.4:4963 , piece_id=YCD5GJFEZMXAGAPZBYNM7SUR3ZDKM3ASKHMS5NEMRDZEDDF6DTGA, seg_pos={0 0}, upload=440669]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; context canceled; EOF Dec 26 13:36:40.826196339 (502h7m40.697279384s) [noise=true , address=45.91.116.4:4963 , piece_id=YCD5GJFEZMXAGAPZBYNM7SUR3ZDKM3ASKHMS5NEMRDZEDDF6DTGA, seg_pos={0 0}, upload=440669]: Long tail cancellation due to optimal threshold Dec 26 13:36:40.826273687 (502h7m40.697356722s) [noise=true , address=45.91.116.12:4253 , piece_id=VOOVGVFPIXWOTCNAZHMVQAEBNAVOGSYDI7GN5W3H3EVW3UJTQ3ZA, seg_pos={0 0}, upload=440669]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gDvXYhShpWvptTU8bivGGwYBTyMF3PJcRv6RJ66vm8aZewejn): protocol: expected piece hash; context canceled; EOF Dec 26 13:36:40.826279708 (502h7m40.697362743s) [noise=true , address=45.91.116.12:4253 , piece_id=VOOVGVFPIXWOTCNAZHMVQAEBNAVOGSYDI7GN5W3H3EVW3UJTQ3ZA, seg_pos={0 0}, upload=440669]: Long tail cancellation due to optimal threshold Dec 26 13:36:40.826308654 (502h7m40.697391689s) [seg_pos={0 0}, upload=440669]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 13:36:40.826430335 (502h7m40.69751337s) [upload=440669]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:36:40.826435234 (502h7m40.697518279s) [upload=440669]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:36:41.158299667 (502h7m41.029382702s) [upload=440669]: Done uploading segments... Dec 26 13:36:41.158324554 (502h7m41.0294076s) [upload=440669]: Upload finished. err: <nil> Duration: 56.114836002s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 32 out of 60 is failed (threshold=6) [X....XXXXXX...XX.....X...XX.X.XXX.XXXXX..X.X.X.X...XXXX.XX.X] duration: 5.000641822s trace: https://jaeger.dev.storj.io/trace/4b02d94968e16f85 FAIL (2025-12-26 13:40:36 UTC) orbiter warning: context deadline exceeded Duration: 5.000662762s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.XXXXXX..X....X..X.X....X......XX.X.X......X.X.XX.X.X..X.XXX] duration: 30.003464787s trace: https://jaeger.dev.storj.io/trace/1637b7f09eb2cdfd FAIL (2025-12-26 13:40:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: 27SNF47PISGU3RPUUHV572Y3OFWA4YG4XLNUHKFOYAWSM6OGNE4A) context deadline exceeded; piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: 7WRBKOIDMOKRLZTKUX2BGQ4O24YYNWLY3PIEGNIUFSKCSORZNMZA) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: SBF2DIUMZU4OKJ556EPZMQY75JJW3I275EBFGH634ZVTX6DEMIGA) context deadline exceeded ======LOGS====== Dec 26 13:39:58.501448607 (502h10m58.372531642s) [upload=440714]: Starting upload Dec 26 13:39:58.501522820 (502h10m58.372605865s) [upload=440714]: Done starting upload Dec 26 13:39:58.501741636 (502h10m58.372824681s) [upload=440714]: Uploading segments... Dec 26 13:39:58.501877545 (502h10m58.37296059s) [upload=440714]: Got next segment. Inline: false Dec 26 13:39:58.501929113 (502h10m58.373012158s) [upload=440714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 13:39:58.501934152 (502h10m58.373017187s) [upload=440714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 13:39:58.756329930 (502h10m58.627412966s) [seg_pos={0 0}, upload=440714]: Begin upload segment... Dec 26 13:39:58.756383793 (502h10m58.627466828s) [seg_pos={0 0}, upload=440714]: Done begin upload segment. Dec 26 13:39:58.756422256 (502h10m58.627505291s) [noise=true , address=145.79.227.3:4150 , piece_id=BNBBHPI2DFYV6KA35ZKK3HAOZFH65HNZEOBR676BPJSV5TZN32JQ, seg_pos={0 0}, upload=440714]: Uploading piece... Dec 26 13:39:58.756465809 (502h10m58.627548844s) [noise=true , address=145.79.227.29:4943 , piece_id=NPQKXTSKCDT4SSC2JF33NEFQ6U66B7BMOL33MQDYHMWCVJNJTZSQ, seg_pos={0 0}, upload=440714]: Uploading piece... Dec 26 13:39:58.757176272 (502h10m58.628259307s) [noise=true , address=77.90.31.18:4808 , piece_id=7WRBKOIDMOKRLZTKUX2BGQ4O24YYNWLY3PIEGNIUFSKCSORZNMZA, seg_pos={0 0}, upload=440714]: Uploading piece... Dec 26 13:39:58.757597153 (502h10m58.628680198s) [noise=true , address=77.90.31.14:4661 , piece_id=4HZ3CBL3VZKRFXX5VYUJ5ZXKK3ERAE4JZOOZN73A2U32P7LTRZAQ, seg_pos={0 0}, upload=440714]: Uploading piece... Dec 26 13:39:58.757952189 (502h10m58.629035234s) [noise=true , address=45.91.116.13:4153 , piece_id=SBF2DIUMZU4OKJ556EPZMQY75JJW3I275EBFGH634ZVTX6DEMIGA, seg_pos={0 0}, upload=440714]: Uploading piece... Dec 26 13:39:58.758301133 (502h10m58.629384168s) [noise=true , address=145.79.227.17:4970 , piece_id=3SWADUXBG54LNRUUZMPAXSGFQRMMETPSV3OSDKAF6DWLYJVJ26DQ, seg_pos={0 0}, upload=440714]: Uploading piece... Dec 26 13:39:58.758469504 (502h10m58.629552539s) [noise=true , address=77.90.31.17:4770 , piece_id=ERS33LNFA7E6GEWKSFXRTUOHTFZWBJFJENXBWLMHDJMQJTQAHMMA, seg_pos={0 0}, upload=440714]: Uploading piece... Dec 26 13:39:58.758757183 (502h10m58.629840218s) [noise=true , address=145.79.227.49:4339 , piece_id=FG2RJGHITCCTNBV2B4CTR7S5KDS4IADTQAVDPS4ND45C76AMHODQ, seg_pos={0 0}, upload=440714]: Uploading piece... Dec 26 13:39:58.758824831 (502h10m58.629907866s) [noise=true , address=45.91.116.14:4191 , piece_id=X65Y5VZGV25ZD6OA3CZ3BJA53USZ4YC5XNMU4TATVCUYZK6TVHBA, seg_pos={0 0}, upload=440714]: Uploading piece... Dec 26 13:39:58.759179637 (502h10m58.630262672s) [noise=true , address=145.79.227.35:4482 , piece_id=IL3WK54QCELQJ6UVFZVE4VZORC47J7IN6QPNISIUSVNMPXRV5USA, seg_pos={0 0}, upload=440714]: Uploading piece... Dec 26 13:39:58.759221356 (502h10m58.630304391s) [noise=true , address=145.79.227.22:4511 , piece_id=QNNZ6JYAKGRYOEIEY2AOCBFTNTX42WTTCUPPFAV6NIKUCFU2JSIQ, seg_pos={0 0}, upload=440714]: Uploading piece... Dec 26 13:39:58.759578286 (502h10m58.630661321s) [noise=true , address=145.79.227.36:4519 , piece_id=NRXJZJOMSFKWXSPCTEXLF3MLUJ7MEMUINY2D33ATI7OJBCSOYICA, seg_pos={0 0}, upload=440714]: Uploading piece... Dec 26 13:39:58.759618853 (502h10m58.630701898s) [noise=true , address=145.79.227.2:4113 , piece_id=TA2UCMMH7W54OEMGS4FFTOVK5PQVDGMM6IIGHCTXULWQZ6YQME4Q, seg_pos={0 0}, upload=440714]: Uploading piece... Dec 26 13:39:58.759901010 (502h10m58.630984045s) [noise=true , address=145.79.227.47:4267 , piece_id=T6MGVOM6R6IYZ5IVQT7CCQG3Y3CDIOA3NEMR5NYQCUMWT4VITNPA, seg_pos={0 0}, upload=440714]: Uploading piece... Dec 26 13:39:58.760055495 (502h10m58.63113854s) [noise=true , address=45.91.116.12:4227 , piece_id=27SNF47PISGU3RPUUHV572Y3OFWA4YG4XLNUHKFOYAWSM6OGNE4A, seg_pos={0 0}, upload=440714]: Uploading piece... Dec 26 13:39:58.760520039 (502h10m58.631603084s) [noise=true , address=145.79.227.44:4194 , piece_id=KW4G5P6W54L7TF7N2IHZWOLXJBHWFSXLEGONUQJEWZRHJUESEYNA, seg_pos={0 0}, upload=440714]: Uploading piece... Dec 26 13:40:00.468550979 (502h11m0.339634024s) [upload=440714]: Next returned nil segment Dec 26 13:40:00.468568223 (502h11m0.339651258s) [upload=440714]: Waiting for error group managing segments... Dec 26 13:40:00.757045617 (502h11m0.628128662s) [noise=true , address=145.79.227.36:4519 , piece_id=NRXJZJOMSFKWXSPCTEXLF3MLUJ7MEMUINY2D33ATI7OJBCSOYICA, seg_pos={0 0}, upload=440714]: Done uploading piece. err: <nil> Dec 26 13:40:00.760374186 (502h11m0.631457221s) [noise=true , address=145.79.227.3:4150 , piece_id=BNBBHPI2DFYV6KA35ZKK3HAOZFH65HNZEOBR676BPJSV5TZN32JQ, seg_pos={0 0}, upload=440714]: Done uploading piece. err: <nil> Dec 26 13:40:00.770890392 (502h11m0.641973427s) [noise=true , address=145.79.227.17:4970 , piece_id=3SWADUXBG54LNRUUZMPAXSGFQRMMETPSV3OSDKAF6DWLYJVJ26DQ, seg_pos={0 0}, upload=440714]: Done uploading piece. err: <nil> Dec 26 13:40:00.775139302 (502h11m0.646222337s) [noise=true , address=145.79.227.49:4339 , piece_id=FG2RJGHITCCTNBV2B4CTR7S5KDS4IADTQAVDPS4ND45C76AMHODQ, seg_pos={0 0}, upload=440714]: Done uploading piece. err: <nil> Dec 26 13:40:00.777589888 (502h11m0.648672923s) [noise=true , address=145.79.227.2:4113 , piece_id=TA2UCMMH7W54OEMGS4FFTOVK5PQVDGMM6IIGHCTXULWQZ6YQME4Q, seg_pos={0 0}, upload=440714]: Done uploading piece. err: <nil> Dec 26 13:40:00.783727976 (502h11m0.654811011s) [noise=true , address=145.79.227.22:4511 , piece_id=QNNZ6JYAKGRYOEIEY2AOCBFTNTX42WTTCUPPFAV6NIKUCFU2JSIQ, seg_pos={0 0}, upload=440714]: Done uploading piece. err: <nil> Dec 26 13:40:00.788505363 (502h11m0.659588409s) [noise=true , address=145.79.227.35:4482 , piece_id=IL3WK54QCELQJ6UVFZVE4VZORC47J7IN6QPNISIUSVNMPXRV5USA, seg_pos={0 0}, upload=440714]: Done uploading piece. err: <nil> Dec 26 13:40:00.792623084 (502h11m0.663706119s) [noise=true , address=145.79.227.29:4943 , piece_id=NPQKXTSKCDT4SSC2JF33NEFQ6U66B7BMOL33MQDYHMWCVJNJTZSQ, seg_pos={0 0}, upload=440714]: Done uploading piece. err: <nil> Dec 26 13:40:00.796828793 (502h11m0.667911838s) [noise=true , address=145.79.227.47:4267 , piece_id=T6MGVOM6R6IYZ5IVQT7CCQG3Y3CDIOA3NEMR5NYQCUMWT4VITNPA, seg_pos={0 0}, upload=440714]: Done uploading piece. err: <nil> Dec 26 13:40:00.984963486 (502h11m0.856046531s) [noise=true , address=45.91.116.13:4153 , piece_id=SBF2DIUMZU4OKJ556EPZMQY75JJW3I275EBFGH634ZVTX6DEMIGA, seg_pos={0 0}, upload=440714]: Done uploading piece. err: <nil> Dec 26 13:40:01.055868132 (502h11m0.926951167s) [noise=true , address=77.90.31.17:4770 , piece_id=ERS33LNFA7E6GEWKSFXRTUOHTFZWBJFJENXBWLMHDJMQJTQAHMMA, seg_pos={0 0}, upload=440714]: Done uploading piece. err: <nil> Dec 26 13:40:01.107676535 (502h11m0.97875957s) [noise=true , address=45.91.116.12:4227 , piece_id=27SNF47PISGU3RPUUHV572Y3OFWA4YG4XLNUHKFOYAWSM6OGNE4A, seg_pos={0 0}, upload=440714]: Done uploading piece. err: <nil> Dec 26 13:40:01.675297812 (502h11m1.546380857s) [noise=true , address=77.90.31.18:4808 , piece_id=7WRBKOIDMOKRLZTKUX2BGQ4O24YYNWLY3PIEGNIUFSKCSORZNMZA, seg_pos={0 0}, upload=440714]: Done uploading piece. err: <nil> Dec 26 13:40:01.675344241 (502h11m1.546427296s) [seg_pos={0 0}, upload=440714]: Segment reached optimal threshold of 13 pieces. Dec 26 13:40:01.675625787 (502h11m1.546708832s) [noise=true , address=145.79.227.44:4194 , piece_id=KW4G5P6W54L7TF7N2IHZWOLXJBHWFSXLEGONUQJEWZRHJUESEYNA, seg_pos={0 0}, upload=440714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 26 13:40:01.675637579 (502h11m1.546720634s) [noise=true , address=145.79.227.44:4194 , piece_id=KW4G5P6W54L7TF7N2IHZWOLXJBHWFSXLEGONUQJEWZRHJUESEYNA, seg_pos={0 0}, upload=440714]: Long tail cancellation due to optimal threshold Dec 26 13:40:01.675916871 (502h11m1.546999916s) [noise=true , address=45.91.116.14:4191 , piece_id=X65Y5VZGV25ZD6OA3CZ3BJA53USZ4YC5XNMU4TATVCUYZK6TVHBA, seg_pos={0 0}, upload=440714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 26 13:40:01.675964903 (502h11m1.547047958s) [noise=true , address=45.91.116.14:4191 , piece_id=X65Y5VZGV25ZD6OA3CZ3BJA53USZ4YC5XNMU4TATVCUYZK6TVHBA, seg_pos={0 0}, upload=440714]: Long tail cancellation due to optimal threshold Dec 26 13:40:01.675986123 (502h11m1.547069168s) [noise=true , address=77.90.31.14:4661 , piece_id=4HZ3CBL3VZKRFXX5VYUJ5ZXKK3ERAE4JZOOZN73A2U32P7LTRZAQ, seg_pos={0 0}, upload=440714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 26 13:40:01.675996633 (502h11m1.547079678s) [noise=true , address=77.90.31.14:4661 , piece_id=4HZ3CBL3VZKRFXX5VYUJ5ZXKK3ERAE4JZOOZN73A2U32P7LTRZAQ, seg_pos={0 0}, upload=440714]: Long tail cancellation due to optimal threshold Dec 26 13:40:01.676027521 (502h11m1.547110566s) [seg_pos={0 0}, upload=440714]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 13:40:01.676132341 (502h11m1.547215396s) [upload=440714]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:40:01.676140217 (502h11m1.547223262s) [upload=440714]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:40:01.994385314 (502h11m1.865468369s) [upload=440714]: Done uploading segments... Dec 26 13:40:01.994413939 (502h11m1.865496984s) [upload=440714]: Upload finished. err: <nil> Duration: 30.003489105s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [.X....XXXXXX...XX.....X...XX.X.XXX.XXXXX..X.X.X.X...XXXX.XX.] duration: 3.697248134s trace: https://jaeger.dev.storj.io/trace/513ace71b29fc3a1 FAIL (2025-12-26 13:40:13 UTC) orbiter warning: context deadline exceeded Duration: 3.6972773s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [..XXXXXX..X....X..X.X....X......XX.X.X......X.X.XX.X.X..X.XX] duration: 30.003411035s trace: https://jaeger.dev.storj.io/trace/7613ddd8eff5ed2d FAIL (2025-12-26 13:39:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: TFQ5LKA7OYGXXM2OPDECHBOZC7456TU3E37MOW3KXTCAETYOAODA) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 4IYKGA45SYX5EX5SB2XJ3JN72HN55UVYIB5HIJLNFYWHLDGTNA2A) context deadline exceeded ======LOGS====== ng segments... Dec 26 13:39:16.575828608 (502h10m16.446911653s) [upload=440705]: Got next segment. Inline: false Dec 26 13:39:16.575854147 (502h10m16.446937192s) [upload=440705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 13:39:16.575858495 (502h10m16.44694153s) [upload=440705]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 13:39:16.824082771 (502h10m16.695165816s) [seg_pos={0 0}, upload=440705]: Begin upload segment... Dec 26 13:39:16.824216125 (502h10m16.69529916s) [seg_pos={0 0}, upload=440705]: Done begin upload segment. Dec 26 13:39:16.824277371 (502h10m16.695360406s) [noise=true , address=145.79.227.51:4410 , piece_id=ESOIMWFXJTFIWVULAH3SWFZMMC67HUKIFA3HYOSQLZRZXAJXANOA, seg_pos={0 0}, upload=440705]: Uploading piece... Dec 26 13:39:16.824296608 (502h10m16.695379643s) [noise=true , address=77.90.31.17:4770 , piece_id=TUTRP5WKBT2Q5E7KZ5PCOFRJZNPADNA4R4FZRVH7TWIU2R4NCAJA, seg_pos={0 0}, upload=440705]: Uploading piece... Dec 26 13:39:16.824919795 (502h10m16.69600283s) [noise=true , address=145.79.227.45:4229 , piece_id=QRXWBHEJFFJKJY3SZ5HNC7PYEZNYCBGGAQ37GPFZEA7SVIXEGUMA, seg_pos={0 0}, upload=440705]: Uploading piece... Dec 26 13:39:16.826116373 (502h10m16.697199408s) [noise=true , address=145.79.227.50:4374 , piece_id=BFJDDRFDNW4PUUDXRHSFMPPDPBLIML2JYXCPQN4QN3YUEYIZMWHA, seg_pos={0 0}, upload=440705]: Uploading piece... Dec 26 13:39:16.826700055 (502h10m16.69778309s) [noise=true , address=145.79.227.42:4123 , piece_id=IF7YPMT35XKLEIBWTBQUPUHHX676LFS4E7UDO73YTTWWV72PYPLQ, seg_pos={0 0}, upload=440705]: Uploading piece... Dec 26 13:39:16.827646988 (502h10m16.698730023s) [noise=true , address=45.91.116.10:4083 , piece_id=T5AR6CZNAYP7K3GFRRLTKRHXJWHKZDCXFT6ANJEJZUL3OFTH7BGQ, seg_pos={0 0}, upload=440705]: Uploading piece... Dec 26 13:39:16.828698399 (502h10m16.699781434s) [noise=true , address=145.79.227.41:4087 , piece_id=2OSC2INV5FM2YJ4TSOZPOFFB5ALJSGXL7ZSV2T4Y7LT2DWW6XYVQ, seg_pos={0 0}, upload=440705]: Uploading piece... Dec 26 13:39:16.829389756 (502h10m16.700472801s) [noise=true , address=145.79.227.2:4113 , piece_id=WPF33QT4GK7IV7U67YHNUNCT6MKXSIQJDJ37V7UQIDHCTGC2JSDQ, seg_pos={0 0}, upload=440705]: Uploading piece... Dec 26 13:39:16.831410203 (502h10m16.702493238s) [noise=true , address=145.79.227.46:4601 , piece_id=7766ZPYSDSAK7GLNS3URJFSSPAK5BZED5ZHJATU75AH36QIUB5QQ, seg_pos={0 0}, upload=440705]: Uploading piece... Dec 26 13:39:16.833302277 (502h10m16.704385322s) [noise=true , address=45.91.116.12:4226 , piece_id=4IYKGA45SYX5EX5SB2XJ3JN72HN55UVYIB5HIJLNFYWHLDGTNA2A, seg_pos={0 0}, upload=440705]: Uploading piece... Dec 26 13:39:16.835960288 (502h10m16.707043333s) [noise=true , address=45.91.116.4:4939 , piece_id=ZQTK6HUIIVHD6KH74HBHSLKDINOVV2GBH64K2F3NJDR3DBITCA2Q, seg_pos={0 0}, upload=440705]: Uploading piece... Dec 26 13:39:16.839204235 (502h10m16.71028728s) [noise=true , address=77.90.31.14:4663 , piece_id=5DX74BODVC6LDSGA2T4ZO5WOBDYBBXT54XY5B3NGXYUNSLW3X2HQ, seg_pos={0 0}, upload=440705]: Uploading piece... Dec 26 13:39:16.839510538 (502h10m16.710593573s) [noise=true , address=145.79.227.35:4481 , piece_id=7YKS26HLACVF4PECWWRDYQBNS6SGWRENRZE62PRX74IBHGOZR5UQ, seg_pos={0 0}, upload=440705]: Uploading piece... Dec 26 13:39:16.841282923 (502h10m16.712365958s) [noise=true , address=45.91.116.7:4010 , piece_id=TFQ5LKA7OYGXXM2OPDECHBOZC7456TU3E37MOW3KXTCAETYOAODA, seg_pos={0 0}, upload=440705]: Uploading piece... Dec 26 13:39:16.841734032 (502h10m16.712817067s) [noise=true , address=145.79.227.16:4910 , piece_id=7SDGQ32CAWDQM3DYRMJM6SKYOSPDI3L5BH54H4WLGCYR3M66ZBKA, seg_pos={0 0}, upload=440705]: Uploading piece... Dec 26 13:39:16.843365659 (502h10m16.714448704s) [noise=true , address=145.79.227.49:4338 , piece_id=ZLDUKXJXMVR2XLQ3TYATFHGKMRKHJIHZUZHAHJIJEAUWSDHI7YWQ, seg_pos={0 0}, upload=440705]: Uploading piece... Dec 26 13:39:18.539029160 (502h10m18.410112195s) [upload=440705]: Next returned nil segment Dec 26 13:39:18.539043328 (502h10m18.410126363s) [upload=440705]: Waiting for error group managing segments... Dec 26 13:39:18.823003861 (502h10m18.694086896s) [noise=true , address=145.79.227.42:4123 , piece_id=IF7YPMT35XKLEIBWTBQUPUHHX676LFS4E7UDO73YTTWWV72PYPLQ, seg_pos={0 0}, upload=440705]: Done uploading piece. err: <nil> Dec 26 13:39:18.831254642 (502h10m18.702337677s) [noise=true , address=145.79.227.50:4374 , piece_id=BFJDDRFDNW4PUUDXRHSFMPPDPBLIML2JYXCPQN4QN3YUEYIZMWHA, seg_pos={0 0}, upload=440705]: Done uploading piece. err: <nil> Dec 26 13:39:18.832837637 (502h10m18.703920682s) [noise=true , address=145.79.227.51:4410 , piece_id=ESOIMWFXJTFIWVULAH3SWFZMMC67HUKIFA3HYOSQLZRZXAJXANOA, seg_pos={0 0}, upload=440705]: Done uploading piece. err: <nil> Dec 26 13:39:18.833451656 (502h10m18.704534691s) [noise=true , address=145.79.227.45:4229 , piece_id=QRXWBHEJFFJKJY3SZ5HNC7PYEZNYCBGGAQ37GPFZEA7SVIXEGUMA, seg_pos={0 0}, upload=440705]: Done uploading piece. err: <nil> Dec 26 13:39:18.836249714 (502h10m18.707332749s) [noise=true , address=145.79.227.41:4087 , piece_id=2OSC2INV5FM2YJ4TSOZPOFFB5ALJSGXL7ZSV2T4Y7LT2DWW6XYVQ, seg_pos={0 0}, upload=440705]: Done uploading piece. err: <nil> Dec 26 13:39:18.836368971 (502h10m18.707452016s) [noise=true , address=145.79.227.2:4113 , piece_id=WPF33QT4GK7IV7U67YHNUNCT6MKXSIQJDJ37V7UQIDHCTGC2JSDQ, seg_pos={0 0}, upload=440705]: Done uploading piece. err: <nil> Dec 26 13:39:18.840025084 (502h10m18.711108119s) [noise=true , address=145.79.227.46:4601 , piece_id=7766ZPYSDSAK7GLNS3URJFSSPAK5BZED5ZHJATU75AH36QIUB5QQ, seg_pos={0 0}, upload=440705]: Done uploading piece. err: <nil> Dec 26 13:39:18.851807820 (502h10m18.722890855s) [noise=true , address=145.79.227.16:4910 , piece_id=7SDGQ32CAWDQM3DYRMJM6SKYOSPDI3L5BH54H4WLGCYR3M66ZBKA, seg_pos={0 0}, upload=440705]: Done uploading piece. err: <nil> Dec 26 13:39:18.867319437 (502h10m18.738402472s) [noise=true , address=145.79.227.35:4481 , piece_id=7YKS26HLACVF4PECWWRDYQBNS6SGWRENRZE62PRX74IBHGOZR5UQ, seg_pos={0 0}, upload=440705]: Done uploading piece. err: <nil> Dec 26 13:39:18.893226098 (502h10m18.764309143s) [noise=true , address=145.79.227.49:4338 , piece_id=ZLDUKXJXMVR2XLQ3TYATFHGKMRKHJIHZUZHAHJIJEAUWSDHI7YWQ, seg_pos={0 0}, upload=440705]: Done uploading piece. err: <nil> Dec 26 13:39:19.102547545 (502h10m18.97363058s) [noise=true , address=45.91.116.12:4226 , piece_id=4IYKGA45SYX5EX5SB2XJ3JN72HN55UVYIB5HIJLNFYWHLDGTNA2A, seg_pos={0 0}, upload=440705]: Done uploading piece. err: <nil> Dec 26 13:39:19.139582737 (502h10m19.010665772s) [noise=true , address=45.91.116.10:4083 , piece_id=T5AR6CZNAYP7K3GFRRLTKRHXJWHKZDCXFT6ANJEJZUL3OFTH7BGQ, seg_pos={0 0}, upload=440705]: Done uploading piece. err: <nil> Dec 26 13:39:19.292982379 (502h10m19.164065424s) [noise=true , address=45.91.116.7:4010 , piece_id=TFQ5LKA7OYGXXM2OPDECHBOZC7456TU3E37MOW3KXTCAETYOAODA, seg_pos={0 0}, upload=440705]: Done uploading piece. err: <nil> Dec 26 13:39:19.293001065 (502h10m19.16408411s) [seg_pos={0 0}, upload=440705]: Segment reached optimal threshold of 13 pieces. Dec 26 13:39:19.293197658 (502h10m19.164280713s) [noise=true , address=45.91.116.4:4939 , piece_id=ZQTK6HUIIVHD6KH74HBHSLKDINOVV2GBH64K2F3NJDR3DBITCA2Q, seg_pos={0 0}, upload=440705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 26 13:39:19.293203770 (502h10m19.164286805s) [noise=true , address=45.91.116.4:4939 , piece_id=ZQTK6HUIIVHD6KH74HBHSLKDINOVV2GBH64K2F3NJDR3DBITCA2Q, seg_pos={0 0}, upload=440705]: Long tail cancellation due to optimal threshold Dec 26 13:39:19.293241422 (502h10m19.164324477s) [noise=true , address=77.90.31.17:4770 , piece_id=TUTRP5WKBT2Q5E7KZ5PCOFRJZNPADNA4R4FZRVH7TWIU2R4NCAJA, seg_pos={0 0}, upload=440705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:54436->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:54436->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54436->77.90.31.17:4770: use of closed network connection Dec 26 13:39:19.293250059 (502h10m19.164333104s) [noise=true , address=77.90.31.17:4770 , piece_id=TUTRP5WKBT2Q5E7KZ5PCOFRJZNPADNA4R4FZRVH7TWIU2R4NCAJA, seg_pos={0 0}, upload=440705]: Long tail cancellation due to optimal threshold Dec 26 13:39:19.293354778 (502h10m19.164437813s) [noise=true , address=77.90.31.14:4663 , piece_id=5DX74BODVC6LDSGA2T4ZO5WOBDYBBXT54XY5B3NGXYUNSLW3X2HQ, seg_pos={0 0}, upload=440705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: write tcp 188.166.225.170:49624->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:49624->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49624->77.90.31.14:4663: use of closed network connection Dec 26 13:39:19.293360178 (502h10m19.164443213s) [noise=true , address=77.90.31.14:4663 , piece_id=5DX74BODVC6LDSGA2T4ZO5WOBDYBBXT54XY5B3NGXYUNSLW3X2HQ, seg_pos={0 0}, upload=440705]: Long tail cancellation due to optimal threshold Dec 26 13:39:19.293377100 (502h10m19.164460135s) [seg_pos={0 0}, upload=440705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 13:39:19.293451122 (502h10m19.164534167s) [upload=440705]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:39:19.293456211 (502h10m19.164539246s) [upload=440705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:39:19.584683172 (502h10m19.455766207s) [upload=440705]: Done uploading segments... Dec 26 13:39:19.584709552 (502h10m19.455792587s) [upload=440705]: Upload finished. err: <nil> Duration: 30.003430954s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [..X....XXXXXX...XX.....X...XX.X.XXX.XXXXX..X.X.X.X...XXXX.XX] duration: 5.001503822s trace: https://jaeger.dev.storj.io/trace/513ace71b29fc3a1 FAIL (2025-12-26 13:39:41 UTC) orbiter warning: context deadline exceeded Duration: 5.001525214s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 17 out of 60 is failed (threshold=3) [X...X..X...................XX.X........X.XXX.X.X..X.X.X.XX..] duration: 48.948881413s trace: https://jaeger.dev.storj.io/trace/175ba4bb6e004432 FAIL (2025-12-26 13:39:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, Piece ID: 2LMZ6SVPT2ERLXTTPQYOFQDONGPSTUKRZPBZC463ZGZDGDQNKUJA) context deadline exceeded; piecestore: (Node ID: 12d7KuNgPYL5Ca2MPKQQCueQNSsYQifHXMhQ87VEXUhrKpLjuks, Piece ID: COXA5OSTIAYWL3C5IQSYUXYNFT7WCMOJ5EYQU5A2OJZMTXN27EWQ) context deadline exceeded; piecestore: (Node ID: 1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, Piece ID: 4KEC7MSAKLWFQ5LG27CWZHZWC4WYUOBVPSLHKZVVDO2HB2V72NIQ) context deadline exceeded; piecestore: (Node ID: 12eaTnUhynF1XeuvMr68SjMyYKM9rLj5tShzpViHcMSjvMs9EBF, Piece ID: 4NSWDLVEHASULNLN4L2SD7YJEOK2HGGSABSZKULLX6NVRACVF6DQ) context deadline exceeded ======LOGS====== Dec 26 13:36:37.629349713 (502h7m37.500432748s) [noise=true , address=77.90.31.13:4653 , piece_id=4NSWDLVEHASULNLN4L2SD7YJEOK2HGGSABSZKULLX6NVRACVF6DQ, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.629469422 (502h7m37.500552457s) [noise=true , address=77.90.31.21:4893 , piece_id=JJKUTOFJLGEALQT2ULTA2SOGJWCQ7RHA7NTTUD7VYQTI5BJS4C4Q, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.629613726 (502h7m37.500696771s) [noise=true , address=145.79.227.44:4223 , piece_id=SGRQEBXPVI5AVILWJMU6ZC4TSQSFQA4YL33BMK6LI3HZ6WNKV2NA, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.629945679 (502h7m37.501028714s) [noise=true , address=145.79.227.34:4915 , piece_id=Y3UN2P5C4JCZVQDPKD6QY5CE7Y2XKAJEJRW75RPAJ5ZZZRO53NIQ, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.630092078 (502h7m37.501175113s) [noise=true , address=145.79.227.30:4780 , piece_id=YPOZFLDB65TFHBXQKXW5B4QT5L7EKINWRPBN2IE4L76DBOLTMSZQ, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.630363495 (502h7m37.50144653s) [noise=true , address=45.91.116.12:4253 , piece_id=VOOVGVFPIXWOTCNAZHMVQAEBNAVOGSYDI7GN5W3H3EVW3UJTQ3ZA, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.630557453 (502h7m37.501640488s) [noise=true , address=45.91.116.9:4294 , piece_id=COXA5OSTIAYWL3C5IQSYUXYNFT7WCMOJ5EYQU5A2OJZMTXN27EWQ, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.630701678 (502h7m37.501784713s) [noise=true , address=64.49.13.169:4716 , piece_id=4YKHMLTI2MIJCADDUM7CMLDPQLJO6BKMXQULZ6RF5HKQ22YW5WJA, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.631024874 (502h7m37.502107919s) [noise=true , address=77.90.31.11:4560 , piece_id=NNZJTUPDWTMVIBRS6Z4UOU6SAA3KGXVCXWROS4YZDIXCEBJLBADA, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.631253629 (502h7m37.502336674s) [noise=true , address=145.79.227.43:4182 , piece_id=QAXA4SBNRU5K2ROO4EK66DACDN26P6Q6VSLNNHPZW2DOSAQHEF2A, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.631552238 (502h7m37.502635273s) [noise=true , address=145.79.227.9:4371 , piece_id=OJMMIDNJ7QTZAGAQD3GKHC57WGTWP4IIVPSXKV7Q252IXT3KNR3Q, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.631811722 (502h7m37.502894767s) [noise=true , address=145.79.227.52:4449 , piece_id=T6MFRO2B27SZYZA35D7CRVIYUGI3PMWDWKGAKO6WRN2UOCKV4RXA, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.632111463 (502h7m37.503194498s) [noise=true , address=145.79.227.51:4434 , piece_id=KHMZXFUE63HDONY5LJUKDR6LHYTEGYBQ5BHWKDKU4X5H5Y64Q6OA, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.632395784 (502h7m37.503478819s) [noise=true , address=77.90.31.10:4532 , piece_id=4KEC7MSAKLWFQ5LG27CWZHZWC4WYUOBVPSLHKZVVDO2HB2V72NIQ, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.634117873 (502h7m37.505200908s) [noise=true , address=45.91.116.4:4963 , piece_id=YCD5GJFEZMXAGAPZBYNM7SUR3ZDKM3ASKHMS5NEMRDZEDDF6DTGA, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.634545508 (502h7m37.505628543s) [noise=true , address=45.91.116.5:4327 , piece_id=ULSWEIDH4LFO5WGT7JIPEL4F7JSQBLWJARZUL44OBNLUC6LOUGOQ, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:39.098109889 (502h7m38.969192935s) [upload=440669]: Next returned nil segment Dec 26 13:36:39.098126871 (502h7m38.969209916s) [upload=440669]: Waiting for error group managing segments... Dec 26 13:36:39.504013736 (502h7m39.375096771s) [noise=true , address=145.79.227.51:4434 , piece_id=KHMZXFUE63HDONY5LJUKDR6LHYTEGYBQ5BHWKDKU4X5H5Y64Q6OA, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:39.513204096 (502h7m39.384287131s) [noise=true , address=145.79.227.52:4449 , piece_id=T6MFRO2B27SZYZA35D7CRVIYUGI3PMWDWKGAKO6WRN2UOCKV4RXA, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:39.532002612 (502h7m39.403085647s) [noise=true , address=145.79.227.43:4182 , piece_id=QAXA4SBNRU5K2ROO4EK66DACDN26P6Q6VSLNNHPZW2DOSAQHEF2A, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:39.591185697 (502h7m39.462268742s) [noise=true , address=145.79.227.9:4371 , piece_id=OJMMIDNJ7QTZAGAQD3GKHC57WGTWP4IIVPSXKV7Q252IXT3KNR3Q, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:39.851571721 (502h7m39.722654766s) [noise=true , address=45.91.116.9:4294 , piece_id=COXA5OSTIAYWL3C5IQSYUXYNFT7WCMOJ5EYQU5A2OJZMTXN27EWQ, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:39.897477856 (502h7m39.768560891s) [noise=true , address=77.90.31.11:4560 , piece_id=NNZJTUPDWTMVIBRS6Z4UOU6SAA3KGXVCXWROS4YZDIXCEBJLBADA, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:40.093563859 (502h7m39.964646905s) [noise=true , address=77.90.31.19:4856 , piece_id=2LMZ6SVPT2ERLXTTPQYOFQDONGPSTUKRZPBZC463ZGZDGDQNKUJA, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:40.406410178 (502h7m40.277493223s) [noise=true , address=145.79.227.44:4223 , piece_id=SGRQEBXPVI5AVILWJMU6ZC4TSQSFQA4YL33BMK6LI3HZ6WNKV2NA, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:40.535693498 (502h7m40.406776543s) [noise=true , address=145.79.227.34:4915 , piece_id=Y3UN2P5C4JCZVQDPKD6QY5CE7Y2XKAJEJRW75RPAJ5ZZZRO53NIQ, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:40.568325205 (502h7m40.43940825s) [noise=true , address=145.79.227.30:4780 , piece_id=YPOZFLDB65TFHBXQKXW5B4QT5L7EKINWRPBN2IE4L76DBOLTMSZQ, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:40.683137940 (502h7m40.554220975s) [noise=true , address=45.91.116.5:4327 , piece_id=ULSWEIDH4LFO5WGT7JIPEL4F7JSQBLWJARZUL44OBNLUC6LOUGOQ, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:40.700510821 (502h7m40.571593856s) [noise=true , address=77.90.31.13:4653 , piece_id=4NSWDLVEHASULNLN4L2SD7YJEOK2HGGSABSZKULLX6NVRACVF6DQ, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:40.794940292 (502h7m40.666023327s) [noise=true , address=77.90.31.10:4532 , piece_id=4KEC7MSAKLWFQ5LG27CWZHZWC4WYUOBVPSLHKZVVDO2HB2V72NIQ, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:40.825821145 (502h7m40.69690419s) [noise=true , address=77.90.31.12:4595 , piece_id=SGG2LCPF3VDF5JAF5PC7RREY35SR5F5AIWOACTLWSQZXLZUMDZKA, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:40.825846833 (502h7m40.696929869s) [seg_pos={0 0}, upload=440669]: Segment reached optimal threshold of 14 pieces. Dec 26 13:36:40.826053557 (502h7m40.697136592s) [noise=true , address=77.90.31.21:4893 , piece_id=JJKUTOFJLGEALQT2ULTA2SOGJWCQ7RHA7NTTUD7VYQTI5BJS4C4Q, seg_pos={0 0}, upload=440669]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC): protocol: expected piece hash; context canceled; EOF Dec 26 13:36:40.826061944 (502h7m40.697144989s) [noise=true , address=77.90.31.21:4893 , piece_id=JJKUTOFJLGEALQT2ULTA2SOGJWCQ7RHA7NTTUD7VYQTI5BJS4C4Q, seg_pos={0 0}, upload=440669]: Long tail cancellation due to optimal threshold Dec 26 13:36:40.826104385 (502h7m40.69718743s) [noise=true , address=64.49.13.169:4716 , piece_id=4YKHMLTI2MIJCADDUM7CMLDPQLJO6BKMXQULZ6RF5HKQ22YW5WJA, seg_pos={0 0}, upload=440669]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:49264->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:49264->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49264->64.49.13.169:4716: use of closed network connection Dec 26 13:36:40.826120344 (502h7m40.69720338s) [noise=true , address=64.49.13.169:4716 , piece_id=4YKHMLTI2MIJCADDUM7CMLDPQLJO6BKMXQULZ6RF5HKQ22YW5WJA, seg_pos={0 0}, upload=440669]: Long tail cancellation due to optimal threshold Dec 26 13:36:40.826190167 (502h7m40.697273202s) [noise=true , address=45.91.116.4:4963 , piece_id=YCD5GJFEZMXAGAPZBYNM7SUR3ZDKM3ASKHMS5NEMRDZEDDF6DTGA, seg_pos={0 0}, upload=440669]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; context canceled; EOF Dec 26 13:36:40.826196339 (502h7m40.697279384s) [noise=true , address=45.91.116.4:4963 , piece_id=YCD5GJFEZMXAGAPZBYNM7SUR3ZDKM3ASKHMS5NEMRDZEDDF6DTGA, seg_pos={0 0}, upload=440669]: Long tail cancellation due to optimal threshold Dec 26 13:36:40.826273687 (502h7m40.697356722s) [noise=true , address=45.91.116.12:4253 , piece_id=VOOVGVFPIXWOTCNAZHMVQAEBNAVOGSYDI7GN5W3H3EVW3UJTQ3ZA, seg_pos={0 0}, upload=440669]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gDvXYhShpWvptTU8bivGGwYBTyMF3PJcRv6RJ66vm8aZewejn): protocol: expected piece hash; context canceled; EOF Dec 26 13:36:40.826279708 (502h7m40.697362743s) [noise=true , address=45.91.116.12:4253 , piece_id=VOOVGVFPIXWOTCNAZHMVQAEBNAVOGSYDI7GN5W3H3EVW3UJTQ3ZA, seg_pos={0 0}, upload=440669]: Long tail cancellation due to optimal threshold Dec 26 13:36:40.826308654 (502h7m40.697391689s) [seg_pos={0 0}, upload=440669]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 13:36:40.826430335 (502h7m40.69751337s) [upload=440669]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:36:40.826435234 (502h7m40.697518279s) [upload=440669]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:36:41.158299667 (502h7m41.029382702s) [upload=440669]: Done uploading segments... Dec 26 13:36:41.158324554 (502h7m41.0294076s) [upload=440669]: Upload finished. err: <nil> Duration: 48.948929174s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X..XXXXXX..X....X..X.X....X......XX.X.X......X.X.XX.X.X..X.X] duration: 30.003844357s trace: https://jaeger.dev.storj.io/trace/69f2caa0b7753892 FAIL (2025-12-26 13:39:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: YEA3NIP2XOKE4M4G5BXEJRA3XWKVZ462AMTEJAEIJ35SPTMPNLMA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: HMSOR3N3SUQLVTAO4FMMIHUOKZHXQ26RESABXAIWFNWHK2TP2DPQ) context deadline exceeded ======LOGS====== d=440697]: Starting upload Dec 26 13:38:39.022656894 (502h9m38.893739959s) [upload=440697]: Done starting upload Dec 26 13:38:39.022809956 (502h9m38.893892991s) [upload=440697]: Uploading segments... Dec 26 13:38:39.022938120 (502h9m38.894021155s) [upload=440697]: Got next segment. Inline: false Dec 26 13:38:39.022971203 (502h9m38.894054238s) [upload=440697]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 13:38:39.022975972 (502h9m38.894059007s) [upload=440697]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 13:38:39.285322119 (502h9m39.156405174s) [seg_pos={0 0}, upload=440697]: Begin upload segment... Dec 26 13:38:39.285370541 (502h9m39.156453586s) [seg_pos={0 0}, upload=440697]: Done begin upload segment. Dec 26 13:38:39.285398744 (502h9m39.156481779s) [noise=true , address=45.91.116.14:4191 , piece_id=YEA3NIP2XOKE4M4G5BXEJRA3XWKVZ462AMTEJAEIJ35SPTMPNLMA, seg_pos={0 0}, upload=440697]: Uploading piece... Dec 26 13:38:39.285423161 (502h9m39.156506196s) [noise=true , address=45.91.116.10:4084 , piece_id=VEAZ3DOQQ5E5ASHKZRPSEIQ22S6JOIYF2EEKMXGEG7BE2BZL2YFA, seg_pos={0 0}, upload=440697]: Uploading piece... Dec 26 13:38:39.286120228 (502h9m39.157203263s) [noise=true , address=145.79.227.3:4151 , piece_id=YD7JBSKNGKLB53EMWPXJHOMTAM6S3OYLV42G4B2HDBOOYOBDLJMQ, seg_pos={0 0}, upload=440697]: Uploading piece... Dec 26 13:38:39.286927505 (502h9m39.15801055s) [noise=true , address=145.79.227.51:4409 , piece_id=MUL72KZYJKZUD6ETMVM37EFQVYUJFQTVVL4AEEQPNDMCVLCM5OBA, seg_pos={0 0}, upload=440697]: Uploading piece... Dec 26 13:38:39.287010274 (502h9m39.158093309s) [noise=true , address=145.79.227.15:4873 , piece_id=DTYAOAYHCGFCOUFPLGGGHBJTZFHCQI3UTLPEUKV6EFXGBEDKCO6Q, seg_pos={0 0}, upload=440697]: Uploading piece... Dec 26 13:38:39.287433600 (502h9m39.158516635s) [noise=true , address=145.79.227.36:4519 , piece_id=LCYMNYB4VOR7UIK4YF5A6ESS46QQEWB7AFOFYTD3NIXXIJ7IIEKQ, seg_pos={0 0}, upload=440697]: Uploading piece... Dec 26 13:38:39.287963088 (502h9m39.159046133s) [noise=true , address=145.79.227.22:4511 , piece_id=4FA22GKBUIRGVY2PCW4PP3Y35KB6M2AMWVXEZA2LSF4DKQUNZJYA, seg_pos={0 0}, upload=440697]: Uploading piece... Dec 26 13:38:39.288292445 (502h9m39.15937548s) [noise=true , address=77.90.31.16:4735 , piece_id=URVWVA6MEB4QERSL3VJHAKGKAKEWOXJ6GVFWDLSWQXNBBSJUB4IQ, seg_pos={0 0}, upload=440697]: Uploading piece... Dec 26 13:38:39.288950257 (502h9m39.160033302s) [noise=true , address=145.79.227.48:4302 , piece_id=6RD2LE4YYJFGUVEESSUVUBX4NU6OH5ASIH7A3LUKTA3LKBF7GQ2Q, seg_pos={0 0}, upload=440697]: Uploading piece... Dec 26 13:38:39.289022175 (502h9m39.16010521s) [noise=true , address=45.91.116.13:4154 , piece_id=LJTWOFPTHRJ6KZFLSPEHMIDV2M2MDKEGSYETBWNZKPUQAKYB2ESQ, seg_pos={0 0}, upload=440697]: Uploading piece... Dec 26 13:38:39.289472822 (502h9m39.160555858s) [noise=true , address=145.79.227.2:4114 , piece_id=X4C4A3PN4UHZALKCKKKC3DYJXEDVJIDAIY2D5FZGDQUP5JXWP6PA, seg_pos={0 0}, upload=440697]: Uploading piece... Dec 26 13:38:39.289922218 (502h9m39.161005263s) [noise=true , address=145.79.227.35:4481 , piece_id=CILDTWJSONJBZLZ2QRTMWPJ7KWYF23IXALSV2NM2R4SFLNGOBLDA, seg_pos={0 0}, upload=440697]: Uploading piece... Dec 26 13:38:39.290443631 (502h9m39.161526676s) [noise=true , address=77.90.31.19:4844 , piece_id=HMSOR3N3SUQLVTAO4FMMIHUOKZHXQ26RESABXAIWFNWHK2TP2DPQ, seg_pos={0 0}, upload=440697]: Uploading piece... Dec 26 13:38:39.290771786 (502h9m39.161854851s) [noise=true , address=77.90.31.21:4880 , piece_id=I5GQ57YXJR4RYWZFNAWVWVKTJG373WWT33PJFZ4HNISHBMUYA6EA, seg_pos={0 0}, upload=440697]: Uploading piece... Dec 26 13:38:39.290880844 (502h9m39.161963909s) [noise=true , address=145.79.227.41:4085 , piece_id=PYGP4ZHWKUHRJWCOJZ27E6R4ARVWRPVZNKWALMCXO3RS4J4IJJJA, seg_pos={0 0}, upload=440697]: Uploading piece... Dec 26 13:38:39.291431613 (502h9m39.162514668s) [noise=true , address=145.79.227.47:4265 , piece_id=4W7QAW2YPV6O6IBNJBYC2CJJPDFOSE5YV3AF2TBTON2OF2DSU5TA, seg_pos={0 0}, upload=440697]: Uploading piece... Dec 26 13:38:41.008688071 (502h9m40.879771116s) [upload=440697]: Next returned nil segment Dec 26 13:38:41.008728328 (502h9m40.879811363s) [upload=440697]: Waiting for error group managing segments... Dec 26 13:38:41.295640573 (502h9m41.166723608s) [noise=true , address=145.79.227.2:4114 , piece_id=X4C4A3PN4UHZALKCKKKC3DYJXEDVJIDAIY2D5FZGDQUP5JXWP6PA, seg_pos={0 0}, upload=440697]: Done uploading piece. err: <nil> Dec 26 13:38:41.296599468 (502h9m41.167682503s) [noise=true , address=145.79.227.35:4481 , piece_id=CILDTWJSONJBZLZ2QRTMWPJ7KWYF23IXALSV2NM2R4SFLNGOBLDA, seg_pos={0 0}, upload=440697]: Done uploading piece. err: <nil> Dec 26 13:38:41.298831579 (502h9m41.169914615s) [noise=true , address=145.79.227.3:4151 , piece_id=YD7JBSKNGKLB53EMWPXJHOMTAM6S3OYLV42G4B2HDBOOYOBDLJMQ, seg_pos={0 0}, upload=440697]: Done uploading piece. err: <nil> Dec 26 13:38:41.300854121 (502h9m41.171937166s) [noise=true , address=145.79.227.51:4409 , piece_id=MUL72KZYJKZUD6ETMVM37EFQVYUJFQTVVL4AEEQPNDMCVLCM5OBA, seg_pos={0 0}, upload=440697]: Done uploading piece. err: <nil> Dec 26 13:38:41.301512585 (502h9m41.17259562s) [noise=true , address=145.79.227.15:4873 , piece_id=DTYAOAYHCGFCOUFPLGGGHBJTZFHCQI3UTLPEUKV6EFXGBEDKCO6Q, seg_pos={0 0}, upload=440697]: Done uploading piece. err: <nil> Dec 26 13:38:41.302468304 (502h9m41.173551339s) [noise=true , address=145.79.227.36:4519 , piece_id=LCYMNYB4VOR7UIK4YF5A6ESS46QQEWB7AFOFYTD3NIXXIJ7IIEKQ, seg_pos={0 0}, upload=440697]: Done uploading piece. err: <nil> Dec 26 13:38:41.306624658 (502h9m41.177707703s) [noise=true , address=145.79.227.48:4302 , piece_id=6RD2LE4YYJFGUVEESSUVUBX4NU6OH5ASIH7A3LUKTA3LKBF7GQ2Q, seg_pos={0 0}, upload=440697]: Done uploading piece. err: <nil> Dec 26 13:38:41.316639649 (502h9m41.187722684s) [noise=true , address=145.79.227.22:4511 , piece_id=4FA22GKBUIRGVY2PCW4PP3Y35KB6M2AMWVXEZA2LSF4DKQUNZJYA, seg_pos={0 0}, upload=440697]: Done uploading piece. err: <nil> Dec 26 13:38:41.317423231 (502h9m41.188506276s) [noise=true , address=145.79.227.41:4085 , piece_id=PYGP4ZHWKUHRJWCOJZ27E6R4ARVWRPVZNKWALMCXO3RS4J4IJJJA, seg_pos={0 0}, upload=440697]: Done uploading piece. err: <nil> Dec 26 13:38:41.329080118 (502h9m41.200163153s) [noise=true , address=145.79.227.47:4265 , piece_id=4W7QAW2YPV6O6IBNJBYC2CJJPDFOSE5YV3AF2TBTON2OF2DSU5TA, seg_pos={0 0}, upload=440697]: Done uploading piece. err: <nil> Dec 26 13:38:41.532829936 (502h9m41.403912981s) [noise=true , address=77.90.31.19:4844 , piece_id=HMSOR3N3SUQLVTAO4FMMIHUOKZHXQ26RESABXAIWFNWHK2TP2DPQ, seg_pos={0 0}, upload=440697]: Done uploading piece. err: <nil> Dec 26 13:38:41.570957338 (502h9m41.442040383s) [noise=true , address=45.91.116.14:4191 , piece_id=YEA3NIP2XOKE4M4G5BXEJRA3XWKVZ462AMTEJAEIJ35SPTMPNLMA, seg_pos={0 0}, upload=440697]: Done uploading piece. err: <nil> Dec 26 13:38:41.588513658 (502h9m41.459596693s) [noise=true , address=77.90.31.21:4880 , piece_id=I5GQ57YXJR4RYWZFNAWVWVKTJG373WWT33PJFZ4HNISHBMUYA6EA, seg_pos={0 0}, upload=440697]: Done uploading piece. err: <nil> Dec 26 13:38:41.588533846 (502h9m41.459616881s) [seg_pos={0 0}, upload=440697]: Segment reached optimal threshold of 13 pieces. Dec 26 13:38:41.588724921 (502h9m41.459807956s) [noise=true , address=45.91.116.10:4084 , piece_id=VEAZ3DOQQ5E5ASHKZRPSEIQ22S6JOIYF2EEKMXGEG7BE2BZL2YFA, seg_pos={0 0}, upload=440697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 26 13:38:41.588731223 (502h9m41.459814268s) [noise=true , address=45.91.116.10:4084 , piece_id=VEAZ3DOQQ5E5ASHKZRPSEIQ22S6JOIYF2EEKMXGEG7BE2BZL2YFA, seg_pos={0 0}, upload=440697]: Long tail cancellation due to optimal threshold Dec 26 13:38:41.589065559 (502h9m41.460148604s) [noise=true , address=45.91.116.13:4154 , piece_id=LJTWOFPTHRJ6KZFLSPEHMIDV2M2MDKEGSYETBWNZKPUQAKYB2ESQ, seg_pos={0 0}, upload=440697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: read tcp 188.166.225.170:59514->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 13:38:41.589073144 (502h9m41.460156179s) [noise=true , address=45.91.116.13:4154 , piece_id=LJTWOFPTHRJ6KZFLSPEHMIDV2M2MDKEGSYETBWNZKPUQAKYB2ESQ, seg_pos={0 0}, upload=440697]: Long tail cancellation due to optimal threshold Dec 26 13:38:41.589094093 (502h9m41.460177138s) [noise=true , address=77.90.31.16:4735 , piece_id=URVWVA6MEB4QERSL3VJHAKGKAKEWOXJ6GVFWDLSWQXNBBSJUB4IQ, seg_pos={0 0}, upload=440697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:46014->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:46014->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46014->77.90.31.16:4735: use of closed network connection Dec 26 13:38:41.589131805 (502h9m41.46021485s) [noise=true , address=77.90.31.16:4735 , piece_id=URVWVA6MEB4QERSL3VJHAKGKAKEWOXJ6GVFWDLSWQXNBBSJUB4IQ, seg_pos={0 0}, upload=440697]: Long tail cancellation due to optimal threshold Dec 26 13:38:41.589164667 (502h9m41.460247713s) [seg_pos={0 0}, upload=440697]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 13:38:41.589273575 (502h9m41.46035662s) [upload=440697]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:38:41.589291830 (502h9m41.460374875s) [upload=440697]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:38:41.880919163 (502h9m41.752002198s) [upload=440697]: Done uploading segments... Dec 26 13:38:41.880937809 (502h9m41.752020844s) [upload=440697]: Upload finished. err: <nil> Duration: 30.003868202s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [X..X....XXXXXX...XX.....X...XX.X.XXX.XXXXX..X.X.X.X...XXXX.X] duration: 5.000823167s trace: https://jaeger.dev.storj.io/trace/6e20e78a436d2ef2 FAIL (2025-12-26 13:38:44 UTC) orbiter warning: context deadline exceeded Duration: 5.00084586s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 18 out of 60 is failed (threshold=3) [XX...X..X...................XX.X........X.XXX.X.X..X.X.X.XX.] duration: 57.350348068s trace: https://jaeger.dev.storj.io/trace/175ba4bb6e004432 FAIL (2025-12-26 13:38:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, Piece ID: 2LMZ6SVPT2ERLXTTPQYOFQDONGPSTUKRZPBZC463ZGZDGDQNKUJA) context deadline exceeded; piecestore: (Node ID: 12d7KuNgPYL5Ca2MPKQQCueQNSsYQifHXMhQ87VEXUhrKpLjuks, Piece ID: COXA5OSTIAYWL3C5IQSYUXYNFT7WCMOJ5EYQU5A2OJZMTXN27EWQ) context deadline exceeded; piecestore: (Node ID: 1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, Piece ID: 4KEC7MSAKLWFQ5LG27CWZHZWC4WYUOBVPSLHKZVVDO2HB2V72NIQ) context deadline exceeded; piecestore: (Node ID: 12eaTnUhynF1XeuvMr68SjMyYKM9rLj5tShzpViHcMSjvMs9EBF, Piece ID: 4NSWDLVEHASULNLN4L2SD7YJEOK2HGGSABSZKULLX6NVRACVF6DQ) context deadline exceeded ======LOGS====== Dec 26 13:36:37.629349713 (502h7m37.500432748s) [noise=true , address=77.90.31.13:4653 , piece_id=4NSWDLVEHASULNLN4L2SD7YJEOK2HGGSABSZKULLX6NVRACVF6DQ, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.629469422 (502h7m37.500552457s) [noise=true , address=77.90.31.21:4893 , piece_id=JJKUTOFJLGEALQT2ULTA2SOGJWCQ7RHA7NTTUD7VYQTI5BJS4C4Q, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.629613726 (502h7m37.500696771s) [noise=true , address=145.79.227.44:4223 , piece_id=SGRQEBXPVI5AVILWJMU6ZC4TSQSFQA4YL33BMK6LI3HZ6WNKV2NA, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.629945679 (502h7m37.501028714s) [noise=true , address=145.79.227.34:4915 , piece_id=Y3UN2P5C4JCZVQDPKD6QY5CE7Y2XKAJEJRW75RPAJ5ZZZRO53NIQ, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.630092078 (502h7m37.501175113s) [noise=true , address=145.79.227.30:4780 , piece_id=YPOZFLDB65TFHBXQKXW5B4QT5L7EKINWRPBN2IE4L76DBOLTMSZQ, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.630363495 (502h7m37.50144653s) [noise=true , address=45.91.116.12:4253 , piece_id=VOOVGVFPIXWOTCNAZHMVQAEBNAVOGSYDI7GN5W3H3EVW3UJTQ3ZA, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.630557453 (502h7m37.501640488s) [noise=true , address=45.91.116.9:4294 , piece_id=COXA5OSTIAYWL3C5IQSYUXYNFT7WCMOJ5EYQU5A2OJZMTXN27EWQ, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.630701678 (502h7m37.501784713s) [noise=true , address=64.49.13.169:4716 , piece_id=4YKHMLTI2MIJCADDUM7CMLDPQLJO6BKMXQULZ6RF5HKQ22YW5WJA, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.631024874 (502h7m37.502107919s) [noise=true , address=77.90.31.11:4560 , piece_id=NNZJTUPDWTMVIBRS6Z4UOU6SAA3KGXVCXWROS4YZDIXCEBJLBADA, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.631253629 (502h7m37.502336674s) [noise=true , address=145.79.227.43:4182 , piece_id=QAXA4SBNRU5K2ROO4EK66DACDN26P6Q6VSLNNHPZW2DOSAQHEF2A, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.631552238 (502h7m37.502635273s) [noise=true , address=145.79.227.9:4371 , piece_id=OJMMIDNJ7QTZAGAQD3GKHC57WGTWP4IIVPSXKV7Q252IXT3KNR3Q, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.631811722 (502h7m37.502894767s) [noise=true , address=145.79.227.52:4449 , piece_id=T6MFRO2B27SZYZA35D7CRVIYUGI3PMWDWKGAKO6WRN2UOCKV4RXA, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.632111463 (502h7m37.503194498s) [noise=true , address=145.79.227.51:4434 , piece_id=KHMZXFUE63HDONY5LJUKDR6LHYTEGYBQ5BHWKDKU4X5H5Y64Q6OA, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.632395784 (502h7m37.503478819s) [noise=true , address=77.90.31.10:4532 , piece_id=4KEC7MSAKLWFQ5LG27CWZHZWC4WYUOBVPSLHKZVVDO2HB2V72NIQ, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.634117873 (502h7m37.505200908s) [noise=true , address=45.91.116.4:4963 , piece_id=YCD5GJFEZMXAGAPZBYNM7SUR3ZDKM3ASKHMS5NEMRDZEDDF6DTGA, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.634545508 (502h7m37.505628543s) [noise=true , address=45.91.116.5:4327 , piece_id=ULSWEIDH4LFO5WGT7JIPEL4F7JSQBLWJARZUL44OBNLUC6LOUGOQ, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:39.098109889 (502h7m38.969192935s) [upload=440669]: Next returned nil segment Dec 26 13:36:39.098126871 (502h7m38.969209916s) [upload=440669]: Waiting for error group managing segments... Dec 26 13:36:39.504013736 (502h7m39.375096771s) [noise=true , address=145.79.227.51:4434 , piece_id=KHMZXFUE63HDONY5LJUKDR6LHYTEGYBQ5BHWKDKU4X5H5Y64Q6OA, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:39.513204096 (502h7m39.384287131s) [noise=true , address=145.79.227.52:4449 , piece_id=T6MFRO2B27SZYZA35D7CRVIYUGI3PMWDWKGAKO6WRN2UOCKV4RXA, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:39.532002612 (502h7m39.403085647s) [noise=true , address=145.79.227.43:4182 , piece_id=QAXA4SBNRU5K2ROO4EK66DACDN26P6Q6VSLNNHPZW2DOSAQHEF2A, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:39.591185697 (502h7m39.462268742s) [noise=true , address=145.79.227.9:4371 , piece_id=OJMMIDNJ7QTZAGAQD3GKHC57WGTWP4IIVPSXKV7Q252IXT3KNR3Q, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:39.851571721 (502h7m39.722654766s) [noise=true , address=45.91.116.9:4294 , piece_id=COXA5OSTIAYWL3C5IQSYUXYNFT7WCMOJ5EYQU5A2OJZMTXN27EWQ, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:39.897477856 (502h7m39.768560891s) [noise=true , address=77.90.31.11:4560 , piece_id=NNZJTUPDWTMVIBRS6Z4UOU6SAA3KGXVCXWROS4YZDIXCEBJLBADA, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:40.093563859 (502h7m39.964646905s) [noise=true , address=77.90.31.19:4856 , piece_id=2LMZ6SVPT2ERLXTTPQYOFQDONGPSTUKRZPBZC463ZGZDGDQNKUJA, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:40.406410178 (502h7m40.277493223s) [noise=true , address=145.79.227.44:4223 , piece_id=SGRQEBXPVI5AVILWJMU6ZC4TSQSFQA4YL33BMK6LI3HZ6WNKV2NA, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:40.535693498 (502h7m40.406776543s) [noise=true , address=145.79.227.34:4915 , piece_id=Y3UN2P5C4JCZVQDPKD6QY5CE7Y2XKAJEJRW75RPAJ5ZZZRO53NIQ, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:40.568325205 (502h7m40.43940825s) [noise=true , address=145.79.227.30:4780 , piece_id=YPOZFLDB65TFHBXQKXW5B4QT5L7EKINWRPBN2IE4L76DBOLTMSZQ, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:40.683137940 (502h7m40.554220975s) [noise=true , address=45.91.116.5:4327 , piece_id=ULSWEIDH4LFO5WGT7JIPEL4F7JSQBLWJARZUL44OBNLUC6LOUGOQ, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:40.700510821 (502h7m40.571593856s) [noise=true , address=77.90.31.13:4653 , piece_id=4NSWDLVEHASULNLN4L2SD7YJEOK2HGGSABSZKULLX6NVRACVF6DQ, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:40.794940292 (502h7m40.666023327s) [noise=true , address=77.90.31.10:4532 , piece_id=4KEC7MSAKLWFQ5LG27CWZHZWC4WYUOBVPSLHKZVVDO2HB2V72NIQ, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:40.825821145 (502h7m40.69690419s) [noise=true , address=77.90.31.12:4595 , piece_id=SGG2LCPF3VDF5JAF5PC7RREY35SR5F5AIWOACTLWSQZXLZUMDZKA, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:40.825846833 (502h7m40.696929869s) [seg_pos={0 0}, upload=440669]: Segment reached optimal threshold of 14 pieces. Dec 26 13:36:40.826053557 (502h7m40.697136592s) [noise=true , address=77.90.31.21:4893 , piece_id=JJKUTOFJLGEALQT2ULTA2SOGJWCQ7RHA7NTTUD7VYQTI5BJS4C4Q, seg_pos={0 0}, upload=440669]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC): protocol: expected piece hash; context canceled; EOF Dec 26 13:36:40.826061944 (502h7m40.697144989s) [noise=true , address=77.90.31.21:4893 , piece_id=JJKUTOFJLGEALQT2ULTA2SOGJWCQ7RHA7NTTUD7VYQTI5BJS4C4Q, seg_pos={0 0}, upload=440669]: Long tail cancellation due to optimal threshold Dec 26 13:36:40.826104385 (502h7m40.69718743s) [noise=true , address=64.49.13.169:4716 , piece_id=4YKHMLTI2MIJCADDUM7CMLDPQLJO6BKMXQULZ6RF5HKQ22YW5WJA, seg_pos={0 0}, upload=440669]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:49264->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:49264->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49264->64.49.13.169:4716: use of closed network connection Dec 26 13:36:40.826120344 (502h7m40.69720338s) [noise=true , address=64.49.13.169:4716 , piece_id=4YKHMLTI2MIJCADDUM7CMLDPQLJO6BKMXQULZ6RF5HKQ22YW5WJA, seg_pos={0 0}, upload=440669]: Long tail cancellation due to optimal threshold Dec 26 13:36:40.826190167 (502h7m40.697273202s) [noise=true , address=45.91.116.4:4963 , piece_id=YCD5GJFEZMXAGAPZBYNM7SUR3ZDKM3ASKHMS5NEMRDZEDDF6DTGA, seg_pos={0 0}, upload=440669]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; context canceled; EOF Dec 26 13:36:40.826196339 (502h7m40.697279384s) [noise=true , address=45.91.116.4:4963 , piece_id=YCD5GJFEZMXAGAPZBYNM7SUR3ZDKM3ASKHMS5NEMRDZEDDF6DTGA, seg_pos={0 0}, upload=440669]: Long tail cancellation due to optimal threshold Dec 26 13:36:40.826273687 (502h7m40.697356722s) [noise=true , address=45.91.116.12:4253 , piece_id=VOOVGVFPIXWOTCNAZHMVQAEBNAVOGSYDI7GN5W3H3EVW3UJTQ3ZA, seg_pos={0 0}, upload=440669]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gDvXYhShpWvptTU8bivGGwYBTyMF3PJcRv6RJ66vm8aZewejn): protocol: expected piece hash; context canceled; EOF Dec 26 13:36:40.826279708 (502h7m40.697362743s) [noise=true , address=45.91.116.12:4253 , piece_id=VOOVGVFPIXWOTCNAZHMVQAEBNAVOGSYDI7GN5W3H3EVW3UJTQ3ZA, seg_pos={0 0}, upload=440669]: Long tail cancellation due to optimal threshold Dec 26 13:36:40.826308654 (502h7m40.697391689s) [seg_pos={0 0}, upload=440669]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 13:36:40.826430335 (502h7m40.69751337s) [upload=440669]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:36:40.826435234 (502h7m40.697518279s) [upload=440669]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:36:41.158299667 (502h7m41.029382702s) [upload=440669]: Done uploading segments... Dec 26 13:36:41.158324554 (502h7m41.0294076s) [upload=440669]: Upload finished. err: <nil> Duration: 57.350378144s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.X..XXXXXX..X....X..X.X....X......XX.X.X......X.X.XX.X.X..X.] duration: 6.818982564s trace: https://jaeger.dev.storj.io/trace/2f76cc85a629954a FAIL (2025-12-26 13:38:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 13:37:33.025305740 (502h8m32.896388775s) [upload=440683]: Starting upload Dec 26 13:37:33.025405800 (502h8m32.896488835s) [upload=440683]: Done starting upload Dec 26 13:37:33.025470342 (502h8m32.896553377s) [upload=440683]: Uploading segments... Dec 26 13:37:33.025945357 (502h8m32.897028392s) [upload=440683]: Got next segment. Inline: false Dec 26 13:37:33.025988669 (502h8m32.897071704s) [upload=440683]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 13:37:33.025995121 (502h8m32.897078156s) [upload=440683]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 13:37:33.275530975 (502h8m33.14661402s) [seg_pos={0 0}, upload=440683]: Begin upload segment... Dec 26 13:37:33.275590518 (502h8m33.146673553s) [seg_pos={0 0}, upload=440683]: Done begin upload segment. Dec 26 13:37:33.275646275 (502h8m33.14672932s) [noise=true , address=145.79.227.12:4765 , piece_id=ETMU6E74XD6PIQ5BNLSVCULV7JGSGI6BXTY32LBXS5VVVWHWAWZA, seg_pos={0 0}, upload=440683]: Uploading piece... Dec 26 13:37:33.275657295 (502h8m33.14674033s) [noise=true , address=45.91.116.6:4975 , piece_id=5IO3ED5QZ2P34FZPHKJWF7ZAEZQOD7FPLTSCJ43BQNN5OCJ3MQHQ, seg_pos={0 0}, upload=440683]: Uploading piece... Dec 26 13:37:33.276351547 (502h8m33.147434612s) [noise=true , address=77.90.31.19:4844 , piece_id=ZXHULQYO4G7F4DQXSGUSCN45AHI7HUOAISJKPA3TSAQ3JRO4PFSA, seg_pos={0 0}, upload=440683]: Uploading piece... Dec 26 13:37:33.276931262 (502h8m33.148014297s) [noise=true , address=145.79.227.46:4601 , piece_id=FXERVAOG7YWTDHPIKWCM2HOVBIVAW4OC2JOZZPDD65SFBHHJ67VA, seg_pos={0 0}, upload=440683]: Uploading piece... Dec 26 13:37:33.277445351 (502h8m33.148528396s) [noise=true , address=77.90.31.21:4878 , piece_id=UNKM33F5B5CD7M7CQ46UXZF547GTZNZ42G3CDQV5L6A2WJ6GTK3A, seg_pos={0 0}, upload=440683]: Uploading piece... Dec 26 13:37:33.277977223 (502h8m33.149060268s) [noise=true , address=145.79.227.48:4302 , piece_id=BDATAJN5ZR3YD3C2HIWYSDR7QXHP6XISFCQ665F4RQR3NLZSPXVA, seg_pos={0 0}, upload=440683]: Uploading piece... Dec 26 13:37:33.278382756 (502h8m33.149465791s) [noise=true , address=45.91.116.9:4264 , piece_id=PMFZHGWVBTRB7XBNGOLDMWWZXEXFFDGDDFT2DTXM2JZU53QWPY5A, seg_pos={0 0}, upload=440683]: Uploading piece... Dec 26 13:37:33.278794599 (502h8m33.149877634s) [noise=true , address=45.91.116.7:4012 , piece_id=4O5B7J5HQT5VQWXDKRAN46XQUKL7DKS4ESOCD7POZCE5MQUQ22HA, seg_pos={0 0}, upload=440683]: Uploading piece... Dec 26 13:37:33.280475701 (502h8m33.151558746s) [noise=true , address=45.91.116.14:4189 , piece_id=VFDRBDBOLJW765TMXAIPPHIRVA5JDFM6PQFL6XEABY57J3DA4CZA, seg_pos={0 0}, upload=440683]: Uploading piece... Dec 26 13:37:33.282597822 (502h8m33.153680857s) [noise=true , address=77.90.31.11:4555 , piece_id=UFO43YM2FH6Q5QAMQTRNVQAM2SG7PRTY3WF4ZUCTGK4E45ODHSIA, seg_pos={0 0}, upload=440683]: Uploading piece... Dec 26 13:37:33.284104430 (502h8m33.155187465s) [noise=true , address=145.79.227.51:4411 , piece_id=RE7DTYNXD2O25FR3PDE65XH2DYAOB6DYRHUV2K4BRJHYTNQTMV3A, seg_pos={0 0}, upload=440683]: Uploading piece... Dec 26 13:37:33.286045537 (502h8m33.157128572s) [noise=true , address=145.79.227.50:4373 , piece_id=6FCOMURGUYFRR7YMTX76QTIIWVNNUZ3CIP2EJDHST557PLHERGUQ, seg_pos={0 0}, upload=440683]: Uploading piece... Dec 26 13:37:33.287765503 (502h8m33.158848548s) [noise=true , address=145.79.227.17:4971 , piece_id=T3I6F37Z6GM5QDAWQSO4CXIWWOEKHMBWS4EQPXQWX46PK6FSWHKQ, seg_pos={0 0}, upload=440683]: Uploading piece... Dec 26 13:37:33.289158976 (502h8m33.160242021s) [noise=true , address=145.79.227.11:4729 , piece_id=4UAXK3PMIV52DOBIHF442EHIU7WJ6YNS7GNIZ65TSVJ7Z45T4TTQ, seg_pos={0 0}, upload=440683]: Uploading piece... Dec 26 13:37:33.290574200 (502h8m33.161657245s) [noise=true , address=145.79.227.20:4438 , piece_id=NUAGAXM4UDFBKXGEKBSU42UMV2S5WVFK46BGAMXO5ELPLJ4PPH3A, seg_pos={0 0}, upload=440683]: Uploading piece... Dec 26 13:37:33.291358414 (502h8m33.162441469s) [noise=true , address=145.79.227.6:4258 , piece_id=5AA4BKT3OYOBCPPMSPATM4REASBMYGLHB2J4NBTC3DT6ZL7ADBBQ, seg_pos={0 0}, upload=440683]: Uploading piece... Dec 26 13:37:34.999691340 (502h8m34.870774385s) [upload=440683]: Next returned nil segment Dec 26 13:37:34.999713061 (502h8m34.870796096s) [upload=440683]: Waiting for error group managing segments... Dec 26 13:37:35.290128486 (502h8m35.161211531s) [noise=true , address=145.79.227.48:4302 , piece_id=BDATAJN5ZR3YD3C2HIWYSDR7QXHP6XISFCQ665F4RQR3NLZSPXVA, seg_pos={0 0}, upload=440683]: Done uploading piece. err: <nil> Dec 26 13:37:35.291035663 (502h8m35.162118708s) [noise=true , address=145.79.227.11:4729 , piece_id=4UAXK3PMIV52DOBIHF442EHIU7WJ6YNS7GNIZ65TSVJ7Z45T4TTQ, seg_pos={0 0}, upload=440683]: Done uploading piece. err: <nil> Dec 26 13:37:35.294320489 (502h8m35.165403534s) [noise=true , address=145.79.227.17:4971 , piece_id=T3I6F37Z6GM5QDAWQSO4CXIWWOEKHMBWS4EQPXQWX46PK6FSWHKQ, seg_pos={0 0}, upload=440683]: Done uploading piece. err: <nil> Dec 26 13:37:35.296386002 (502h8m35.167469037s) [noise=true , address=145.79.227.51:4411 , piece_id=RE7DTYNXD2O25FR3PDE65XH2DYAOB6DYRHUV2K4BRJHYTNQTMV3A, seg_pos={0 0}, upload=440683]: Done uploading piece. err: <nil> Dec 26 13:37:35.297374585 (502h8m35.16845762s) [noise=true , address=145.79.227.6:4258 , piece_id=5AA4BKT3OYOBCPPMSPATM4REASBMYGLHB2J4NBTC3DT6ZL7ADBBQ, seg_pos={0 0}, upload=440683]: Done uploading piece. err: <nil> Dec 26 13:37:35.310126666 (502h8m35.181209701s) [noise=true , address=145.79.227.46:4601 , piece_id=FXERVAOG7YWTDHPIKWCM2HOVBIVAW4OC2JOZZPDD65SFBHHJ67VA, seg_pos={0 0}, upload=440683]: Done uploading piece. err: <nil> Dec 26 13:37:35.314770780 (502h8m35.185853825s) [noise=true , address=145.79.227.50:4373 , piece_id=6FCOMURGUYFRR7YMTX76QTIIWVNNUZ3CIP2EJDHST557PLHERGUQ, seg_pos={0 0}, upload=440683]: Done uploading piece. err: <nil> Dec 26 13:37:35.326006024 (502h8m35.197089079s) [noise=true , address=145.79.227.20:4438 , piece_id=NUAGAXM4UDFBKXGEKBSU42UMV2S5WVFK46BGAMXO5ELPLJ4PPH3A, seg_pos={0 0}, upload=440683]: Done uploading piece. err: <nil> Dec 26 13:37:35.326111044 (502h8m35.197194109s) [noise=true , address=145.79.227.12:4765 , piece_id=ETMU6E74XD6PIQ5BNLSVCULV7JGSGI6BXTY32LBXS5VVVWHWAWZA, seg_pos={0 0}, upload=440683]: Done uploading piece. err: <nil> Dec 26 13:37:35.537498004 (502h8m35.408581039s) [noise=true , address=45.91.116.7:4012 , piece_id=4O5B7J5HQT5VQWXDKRAN46XQUKL7DKS4ESOCD7POZCE5MQUQ22HA, seg_pos={0 0}, upload=440683]: Done uploading piece. err: <nil> Dec 26 13:37:35.839234778 (502h8m35.710317813s) [noise=true , address=45.91.116.14:4189 , piece_id=VFDRBDBOLJW765TMXAIPPHIRVA5JDFM6PQFL6XEABY57J3DA4CZA, seg_pos={0 0}, upload=440683]: Done uploading piece. err: <nil> Dec 26 13:37:36.773075325 (502h8m36.64415836s) [noise=true , address=77.90.31.21:4878 , piece_id=UNKM33F5B5CD7M7CQ46UXZF547GTZNZ42G3CDQV5L6A2WJ6GTK3A, seg_pos={0 0}, upload=440683]: Done uploading piece. err: <nil> Dec 26 13:37:36.873310011 (502h8m36.744393056s) [noise=true , address=45.91.116.6:4975 , piece_id=5IO3ED5QZ2P34FZPHKJWF7ZAEZQOD7FPLTSCJ43BQNN5OCJ3MQHQ, seg_pos={0 0}, upload=440683]: Done uploading piece. err: <nil> Dec 26 13:37:36.873331542 (502h8m36.744414578s) [seg_pos={0 0}, upload=440683]: Segment reached optimal threshold of 13 pieces. Dec 26 13:37:36.873547834 (502h8m36.744630879s) [noise=true , address=45.91.116.9:4264 , piece_id=PMFZHGWVBTRB7XBNGOLDMWWZXEXFFDGDDFT2DTXM2JZU53QWPY5A, seg_pos={0 0}, upload=440683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 13:37:36.873557041 (502h8m36.744640076s) [noise=true , address=45.91.116.9:4264 , piece_id=PMFZHGWVBTRB7XBNGOLDMWWZXEXFFDGDDFT2DTXM2JZU53QWPY5A, seg_pos={0 0}, upload=440683]: Long tail cancellation due to optimal threshold Dec 26 13:37:36.873613529 (502h8m36.744696594s) [noise=true , address=77.90.31.11:4555 , piece_id=UFO43YM2FH6Q5QAMQTRNVQAM2SG7PRTY3WF4ZUCTGK4E45ODHSIA, seg_pos={0 0}, upload=440683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: write tcp 188.166.225.170:48274->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:48274->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48274->77.90.31.11:4555: use of closed network connection Dec 26 13:37:36.873624169 (502h8m36.744707204s) [noise=true , address=77.90.31.11:4555 , piece_id=UFO43YM2FH6Q5QAMQTRNVQAM2SG7PRTY3WF4ZUCTGK4E45ODHSIA, seg_pos={0 0}, upload=440683]: Long tail cancellation due to optimal threshold Dec 26 13:37:36.873838577 (502h8m36.744921643s) [noise=true , address=77.90.31.19:4844 , piece_id=ZXHULQYO4G7F4DQXSGUSCN45AHI7HUOAISJKPA3TSAQ3JRO4PFSA, seg_pos={0 0}, upload=440683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 26 13:37:36.873850080 (502h8m36.744933115s) [noise=true , address=77.90.31.19:4844 , piece_id=ZXHULQYO4G7F4DQXSGUSCN45AHI7HUOAISJKPA3TSAQ3JRO4PFSA, seg_pos={0 0}, upload=440683]: Long tail cancellation due to optimal threshold Dec 26 13:37:36.873873855 (502h8m36.7449569s) [seg_pos={0 0}, upload=440683]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 13:37:36.873949849 (502h8m36.745032884s) [upload=440683]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:37:36.873955571 (502h8m36.745038606s) [upload=440683]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:37:37.163799609 (502h8m37.034882644s) [upload=440683]: Done uploading segments... Dec 26 13:37:37.163820018 (502h8m37.034903053s) [upload=440683]: Upload finished. err: <nil> Duration: 6.81901219s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [.X..X....XXXXXX...XX.....X...XX.X.XXX.XXXXX..X.X.X.X...XXXX.] duration: 3.39742731s trace: https://jaeger.dev.storj.io/trace/4b9683e993ef9f16 FAIL (2025-12-26 13:38:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.397526629s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..X..XXXXXX..X....X..X.X....X......XX.X.X......X.X.XX.X.X..X] duration: 30.003870239s trace: https://jaeger.dev.storj.io/trace/2f76cc85a629954a FAIL (2025-12-26 13:38:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 13:37:33.025305740 (502h8m32.896388775s) [upload=440683]: Starting upload Dec 26 13:37:33.025405800 (502h8m32.896488835s) [upload=440683]: Done starting upload Dec 26 13:37:33.025470342 (502h8m32.896553377s) [upload=440683]: Uploading segments... Dec 26 13:37:33.025945357 (502h8m32.897028392s) [upload=440683]: Got next segment. Inline: false Dec 26 13:37:33.025988669 (502h8m32.897071704s) [upload=440683]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 13:37:33.025995121 (502h8m32.897078156s) [upload=440683]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 13:37:33.275530975 (502h8m33.14661402s) [seg_pos={0 0}, upload=440683]: Begin upload segment... Dec 26 13:37:33.275590518 (502h8m33.146673553s) [seg_pos={0 0}, upload=440683]: Done begin upload segment. Dec 26 13:37:33.275646275 (502h8m33.14672932s) [noise=true , address=145.79.227.12:4765 , piece_id=ETMU6E74XD6PIQ5BNLSVCULV7JGSGI6BXTY32LBXS5VVVWHWAWZA, seg_pos={0 0}, upload=440683]: Uploading piece... Dec 26 13:37:33.275657295 (502h8m33.14674033s) [noise=true , address=45.91.116.6:4975 , piece_id=5IO3ED5QZ2P34FZPHKJWF7ZAEZQOD7FPLTSCJ43BQNN5OCJ3MQHQ, seg_pos={0 0}, upload=440683]: Uploading piece... Dec 26 13:37:33.276351547 (502h8m33.147434612s) [noise=true , address=77.90.31.19:4844 , piece_id=ZXHULQYO4G7F4DQXSGUSCN45AHI7HUOAISJKPA3TSAQ3JRO4PFSA, seg_pos={0 0}, upload=440683]: Uploading piece... Dec 26 13:37:33.276931262 (502h8m33.148014297s) [noise=true , address=145.79.227.46:4601 , piece_id=FXERVAOG7YWTDHPIKWCM2HOVBIVAW4OC2JOZZPDD65SFBHHJ67VA, seg_pos={0 0}, upload=440683]: Uploading piece... Dec 26 13:37:33.277445351 (502h8m33.148528396s) [noise=true , address=77.90.31.21:4878 , piece_id=UNKM33F5B5CD7M7CQ46UXZF547GTZNZ42G3CDQV5L6A2WJ6GTK3A, seg_pos={0 0}, upload=440683]: Uploading piece... Dec 26 13:37:33.277977223 (502h8m33.149060268s) [noise=true , address=145.79.227.48:4302 , piece_id=BDATAJN5ZR3YD3C2HIWYSDR7QXHP6XISFCQ665F4RQR3NLZSPXVA, seg_pos={0 0}, upload=440683]: Uploading piece... Dec 26 13:37:33.278382756 (502h8m33.149465791s) [noise=true , address=45.91.116.9:4264 , piece_id=PMFZHGWVBTRB7XBNGOLDMWWZXEXFFDGDDFT2DTXM2JZU53QWPY5A, seg_pos={0 0}, upload=440683]: Uploading piece... Dec 26 13:37:33.278794599 (502h8m33.149877634s) [noise=true , address=45.91.116.7:4012 , piece_id=4O5B7J5HQT5VQWXDKRAN46XQUKL7DKS4ESOCD7POZCE5MQUQ22HA, seg_pos={0 0}, upload=440683]: Uploading piece... Dec 26 13:37:33.280475701 (502h8m33.151558746s) [noise=true , address=45.91.116.14:4189 , piece_id=VFDRBDBOLJW765TMXAIPPHIRVA5JDFM6PQFL6XEABY57J3DA4CZA, seg_pos={0 0}, upload=440683]: Uploading piece... Dec 26 13:37:33.282597822 (502h8m33.153680857s) [noise=true , address=77.90.31.11:4555 , piece_id=UFO43YM2FH6Q5QAMQTRNVQAM2SG7PRTY3WF4ZUCTGK4E45ODHSIA, seg_pos={0 0}, upload=440683]: Uploading piece... Dec 26 13:37:33.284104430 (502h8m33.155187465s) [noise=true , address=145.79.227.51:4411 , piece_id=RE7DTYNXD2O25FR3PDE65XH2DYAOB6DYRHUV2K4BRJHYTNQTMV3A, seg_pos={0 0}, upload=440683]: Uploading piece... Dec 26 13:37:33.286045537 (502h8m33.157128572s) [noise=true , address=145.79.227.50:4373 , piece_id=6FCOMURGUYFRR7YMTX76QTIIWVNNUZ3CIP2EJDHST557PLHERGUQ, seg_pos={0 0}, upload=440683]: Uploading piece... Dec 26 13:37:33.287765503 (502h8m33.158848548s) [noise=true , address=145.79.227.17:4971 , piece_id=T3I6F37Z6GM5QDAWQSO4CXIWWOEKHMBWS4EQPXQWX46PK6FSWHKQ, seg_pos={0 0}, upload=440683]: Uploading piece... Dec 26 13:37:33.289158976 (502h8m33.160242021s) [noise=true , address=145.79.227.11:4729 , piece_id=4UAXK3PMIV52DOBIHF442EHIU7WJ6YNS7GNIZ65TSVJ7Z45T4TTQ, seg_pos={0 0}, upload=440683]: Uploading piece... Dec 26 13:37:33.290574200 (502h8m33.161657245s) [noise=true , address=145.79.227.20:4438 , piece_id=NUAGAXM4UDFBKXGEKBSU42UMV2S5WVFK46BGAMXO5ELPLJ4PPH3A, seg_pos={0 0}, upload=440683]: Uploading piece... Dec 26 13:37:33.291358414 (502h8m33.162441469s) [noise=true , address=145.79.227.6:4258 , piece_id=5AA4BKT3OYOBCPPMSPATM4REASBMYGLHB2J4NBTC3DT6ZL7ADBBQ, seg_pos={0 0}, upload=440683]: Uploading piece... Dec 26 13:37:34.999691340 (502h8m34.870774385s) [upload=440683]: Next returned nil segment Dec 26 13:37:34.999713061 (502h8m34.870796096s) [upload=440683]: Waiting for error group managing segments... Dec 26 13:37:35.290128486 (502h8m35.161211531s) [noise=true , address=145.79.227.48:4302 , piece_id=BDATAJN5ZR3YD3C2HIWYSDR7QXHP6XISFCQ665F4RQR3NLZSPXVA, seg_pos={0 0}, upload=440683]: Done uploading piece. err: <nil> Dec 26 13:37:35.291035663 (502h8m35.162118708s) [noise=true , address=145.79.227.11:4729 , piece_id=4UAXK3PMIV52DOBIHF442EHIU7WJ6YNS7GNIZ65TSVJ7Z45T4TTQ, seg_pos={0 0}, upload=440683]: Done uploading piece. err: <nil> Dec 26 13:37:35.294320489 (502h8m35.165403534s) [noise=true , address=145.79.227.17:4971 , piece_id=T3I6F37Z6GM5QDAWQSO4CXIWWOEKHMBWS4EQPXQWX46PK6FSWHKQ, seg_pos={0 0}, upload=440683]: Done uploading piece. err: <nil> Dec 26 13:37:35.296386002 (502h8m35.167469037s) [noise=true , address=145.79.227.51:4411 , piece_id=RE7DTYNXD2O25FR3PDE65XH2DYAOB6DYRHUV2K4BRJHYTNQTMV3A, seg_pos={0 0}, upload=440683]: Done uploading piece. err: <nil> Dec 26 13:37:35.297374585 (502h8m35.16845762s) [noise=true , address=145.79.227.6:4258 , piece_id=5AA4BKT3OYOBCPPMSPATM4REASBMYGLHB2J4NBTC3DT6ZL7ADBBQ, seg_pos={0 0}, upload=440683]: Done uploading piece. err: <nil> Dec 26 13:37:35.310126666 (502h8m35.181209701s) [noise=true , address=145.79.227.46:4601 , piece_id=FXERVAOG7YWTDHPIKWCM2HOVBIVAW4OC2JOZZPDD65SFBHHJ67VA, seg_pos={0 0}, upload=440683]: Done uploading piece. err: <nil> Dec 26 13:37:35.314770780 (502h8m35.185853825s) [noise=true , address=145.79.227.50:4373 , piece_id=6FCOMURGUYFRR7YMTX76QTIIWVNNUZ3CIP2EJDHST557PLHERGUQ, seg_pos={0 0}, upload=440683]: Done uploading piece. err: <nil> Dec 26 13:37:35.326006024 (502h8m35.197089079s) [noise=true , address=145.79.227.20:4438 , piece_id=NUAGAXM4UDFBKXGEKBSU42UMV2S5WVFK46BGAMXO5ELPLJ4PPH3A, seg_pos={0 0}, upload=440683]: Done uploading piece. err: <nil> Dec 26 13:37:35.326111044 (502h8m35.197194109s) [noise=true , address=145.79.227.12:4765 , piece_id=ETMU6E74XD6PIQ5BNLSVCULV7JGSGI6BXTY32LBXS5VVVWHWAWZA, seg_pos={0 0}, upload=440683]: Done uploading piece. err: <nil> Dec 26 13:37:35.537498004 (502h8m35.408581039s) [noise=true , address=45.91.116.7:4012 , piece_id=4O5B7J5HQT5VQWXDKRAN46XQUKL7DKS4ESOCD7POZCE5MQUQ22HA, seg_pos={0 0}, upload=440683]: Done uploading piece. err: <nil> Dec 26 13:37:35.839234778 (502h8m35.710317813s) [noise=true , address=45.91.116.14:4189 , piece_id=VFDRBDBOLJW765TMXAIPPHIRVA5JDFM6PQFL6XEABY57J3DA4CZA, seg_pos={0 0}, upload=440683]: Done uploading piece. err: <nil> Dec 26 13:37:36.773075325 (502h8m36.64415836s) [noise=true , address=77.90.31.21:4878 , piece_id=UNKM33F5B5CD7M7CQ46UXZF547GTZNZ42G3CDQV5L6A2WJ6GTK3A, seg_pos={0 0}, upload=440683]: Done uploading piece. err: <nil> Dec 26 13:37:36.873310011 (502h8m36.744393056s) [noise=true , address=45.91.116.6:4975 , piece_id=5IO3ED5QZ2P34FZPHKJWF7ZAEZQOD7FPLTSCJ43BQNN5OCJ3MQHQ, seg_pos={0 0}, upload=440683]: Done uploading piece. err: <nil> Dec 26 13:37:36.873331542 (502h8m36.744414578s) [seg_pos={0 0}, upload=440683]: Segment reached optimal threshold of 13 pieces. Dec 26 13:37:36.873547834 (502h8m36.744630879s) [noise=true , address=45.91.116.9:4264 , piece_id=PMFZHGWVBTRB7XBNGOLDMWWZXEXFFDGDDFT2DTXM2JZU53QWPY5A, seg_pos={0 0}, upload=440683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 13:37:36.873557041 (502h8m36.744640076s) [noise=true , address=45.91.116.9:4264 , piece_id=PMFZHGWVBTRB7XBNGOLDMWWZXEXFFDGDDFT2DTXM2JZU53QWPY5A, seg_pos={0 0}, upload=440683]: Long tail cancellation due to optimal threshold Dec 26 13:37:36.873613529 (502h8m36.744696594s) [noise=true , address=77.90.31.11:4555 , piece_id=UFO43YM2FH6Q5QAMQTRNVQAM2SG7PRTY3WF4ZUCTGK4E45ODHSIA, seg_pos={0 0}, upload=440683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: write tcp 188.166.225.170:48274->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:48274->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48274->77.90.31.11:4555: use of closed network connection Dec 26 13:37:36.873624169 (502h8m36.744707204s) [noise=true , address=77.90.31.11:4555 , piece_id=UFO43YM2FH6Q5QAMQTRNVQAM2SG7PRTY3WF4ZUCTGK4E45ODHSIA, seg_pos={0 0}, upload=440683]: Long tail cancellation due to optimal threshold Dec 26 13:37:36.873838577 (502h8m36.744921643s) [noise=true , address=77.90.31.19:4844 , piece_id=ZXHULQYO4G7F4DQXSGUSCN45AHI7HUOAISJKPA3TSAQ3JRO4PFSA, seg_pos={0 0}, upload=440683]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 26 13:37:36.873850080 (502h8m36.744933115s) [noise=true , address=77.90.31.19:4844 , piece_id=ZXHULQYO4G7F4DQXSGUSCN45AHI7HUOAISJKPA3TSAQ3JRO4PFSA, seg_pos={0 0}, upload=440683]: Long tail cancellation due to optimal threshold Dec 26 13:37:36.873873855 (502h8m36.7449569s) [seg_pos={0 0}, upload=440683]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 13:37:36.873949849 (502h8m36.745032884s) [upload=440683]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:37:36.873955571 (502h8m36.745038606s) [upload=440683]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:37:37.163799609 (502h8m37.034882644s) [upload=440683]: Done uploading segments... Dec 26 13:37:37.163820018 (502h8m37.034903053s) [upload=440683]: Upload finished. err: <nil> Duration: 30.003894065s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 19 out of 60 is failed (threshold=3) [XXX...X..X...................XX.X........X.XXX.X.X..X.X.X.XX] duration: 1m0.002871125s trace: https://jaeger.dev.storj.io/trace/175ba4bb6e004432 FAIL (2025-12-26 13:37:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, Piece ID: 2LMZ6SVPT2ERLXTTPQYOFQDONGPSTUKRZPBZC463ZGZDGDQNKUJA) context deadline exceeded; piecestore: (Node ID: 12d7KuNgPYL5Ca2MPKQQCueQNSsYQifHXMhQ87VEXUhrKpLjuks, Piece ID: COXA5OSTIAYWL3C5IQSYUXYNFT7WCMOJ5EYQU5A2OJZMTXN27EWQ) context deadline exceeded; piecestore: (Node ID: 1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, Piece ID: 4KEC7MSAKLWFQ5LG27CWZHZWC4WYUOBVPSLHKZVVDO2HB2V72NIQ) context deadline exceeded; piecestore: (Node ID: 12eaTnUhynF1XeuvMr68SjMyYKM9rLj5tShzpViHcMSjvMs9EBF, Piece ID: 4NSWDLVEHASULNLN4L2SD7YJEOK2HGGSABSZKULLX6NVRACVF6DQ) context deadline exceeded ======LOGS====== Dec 26 13:36:37.629349713 (502h7m37.500432748s) [noise=true , address=77.90.31.13:4653 , piece_id=4NSWDLVEHASULNLN4L2SD7YJEOK2HGGSABSZKULLX6NVRACVF6DQ, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.629469422 (502h7m37.500552457s) [noise=true , address=77.90.31.21:4893 , piece_id=JJKUTOFJLGEALQT2ULTA2SOGJWCQ7RHA7NTTUD7VYQTI5BJS4C4Q, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.629613726 (502h7m37.500696771s) [noise=true , address=145.79.227.44:4223 , piece_id=SGRQEBXPVI5AVILWJMU6ZC4TSQSFQA4YL33BMK6LI3HZ6WNKV2NA, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.629945679 (502h7m37.501028714s) [noise=true , address=145.79.227.34:4915 , piece_id=Y3UN2P5C4JCZVQDPKD6QY5CE7Y2XKAJEJRW75RPAJ5ZZZRO53NIQ, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.630092078 (502h7m37.501175113s) [noise=true , address=145.79.227.30:4780 , piece_id=YPOZFLDB65TFHBXQKXW5B4QT5L7EKINWRPBN2IE4L76DBOLTMSZQ, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.630363495 (502h7m37.50144653s) [noise=true , address=45.91.116.12:4253 , piece_id=VOOVGVFPIXWOTCNAZHMVQAEBNAVOGSYDI7GN5W3H3EVW3UJTQ3ZA, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.630557453 (502h7m37.501640488s) [noise=true , address=45.91.116.9:4294 , piece_id=COXA5OSTIAYWL3C5IQSYUXYNFT7WCMOJ5EYQU5A2OJZMTXN27EWQ, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.630701678 (502h7m37.501784713s) [noise=true , address=64.49.13.169:4716 , piece_id=4YKHMLTI2MIJCADDUM7CMLDPQLJO6BKMXQULZ6RF5HKQ22YW5WJA, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.631024874 (502h7m37.502107919s) [noise=true , address=77.90.31.11:4560 , piece_id=NNZJTUPDWTMVIBRS6Z4UOU6SAA3KGXVCXWROS4YZDIXCEBJLBADA, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.631253629 (502h7m37.502336674s) [noise=true , address=145.79.227.43:4182 , piece_id=QAXA4SBNRU5K2ROO4EK66DACDN26P6Q6VSLNNHPZW2DOSAQHEF2A, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.631552238 (502h7m37.502635273s) [noise=true , address=145.79.227.9:4371 , piece_id=OJMMIDNJ7QTZAGAQD3GKHC57WGTWP4IIVPSXKV7Q252IXT3KNR3Q, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.631811722 (502h7m37.502894767s) [noise=true , address=145.79.227.52:4449 , piece_id=T6MFRO2B27SZYZA35D7CRVIYUGI3PMWDWKGAKO6WRN2UOCKV4RXA, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.632111463 (502h7m37.503194498s) [noise=true , address=145.79.227.51:4434 , piece_id=KHMZXFUE63HDONY5LJUKDR6LHYTEGYBQ5BHWKDKU4X5H5Y64Q6OA, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.632395784 (502h7m37.503478819s) [noise=true , address=77.90.31.10:4532 , piece_id=4KEC7MSAKLWFQ5LG27CWZHZWC4WYUOBVPSLHKZVVDO2HB2V72NIQ, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.634117873 (502h7m37.505200908s) [noise=true , address=45.91.116.4:4963 , piece_id=YCD5GJFEZMXAGAPZBYNM7SUR3ZDKM3ASKHMS5NEMRDZEDDF6DTGA, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:37.634545508 (502h7m37.505628543s) [noise=true , address=45.91.116.5:4327 , piece_id=ULSWEIDH4LFO5WGT7JIPEL4F7JSQBLWJARZUL44OBNLUC6LOUGOQ, seg_pos={0 0}, upload=440669]: Uploading piece... Dec 26 13:36:39.098109889 (502h7m38.969192935s) [upload=440669]: Next returned nil segment Dec 26 13:36:39.098126871 (502h7m38.969209916s) [upload=440669]: Waiting for error group managing segments... Dec 26 13:36:39.504013736 (502h7m39.375096771s) [noise=true , address=145.79.227.51:4434 , piece_id=KHMZXFUE63HDONY5LJUKDR6LHYTEGYBQ5BHWKDKU4X5H5Y64Q6OA, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:39.513204096 (502h7m39.384287131s) [noise=true , address=145.79.227.52:4449 , piece_id=T6MFRO2B27SZYZA35D7CRVIYUGI3PMWDWKGAKO6WRN2UOCKV4RXA, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:39.532002612 (502h7m39.403085647s) [noise=true , address=145.79.227.43:4182 , piece_id=QAXA4SBNRU5K2ROO4EK66DACDN26P6Q6VSLNNHPZW2DOSAQHEF2A, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:39.591185697 (502h7m39.462268742s) [noise=true , address=145.79.227.9:4371 , piece_id=OJMMIDNJ7QTZAGAQD3GKHC57WGTWP4IIVPSXKV7Q252IXT3KNR3Q, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:39.851571721 (502h7m39.722654766s) [noise=true , address=45.91.116.9:4294 , piece_id=COXA5OSTIAYWL3C5IQSYUXYNFT7WCMOJ5EYQU5A2OJZMTXN27EWQ, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:39.897477856 (502h7m39.768560891s) [noise=true , address=77.90.31.11:4560 , piece_id=NNZJTUPDWTMVIBRS6Z4UOU6SAA3KGXVCXWROS4YZDIXCEBJLBADA, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:40.093563859 (502h7m39.964646905s) [noise=true , address=77.90.31.19:4856 , piece_id=2LMZ6SVPT2ERLXTTPQYOFQDONGPSTUKRZPBZC463ZGZDGDQNKUJA, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:40.406410178 (502h7m40.277493223s) [noise=true , address=145.79.227.44:4223 , piece_id=SGRQEBXPVI5AVILWJMU6ZC4TSQSFQA4YL33BMK6LI3HZ6WNKV2NA, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:40.535693498 (502h7m40.406776543s) [noise=true , address=145.79.227.34:4915 , piece_id=Y3UN2P5C4JCZVQDPKD6QY5CE7Y2XKAJEJRW75RPAJ5ZZZRO53NIQ, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:40.568325205 (502h7m40.43940825s) [noise=true , address=145.79.227.30:4780 , piece_id=YPOZFLDB65TFHBXQKXW5B4QT5L7EKINWRPBN2IE4L76DBOLTMSZQ, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:40.683137940 (502h7m40.554220975s) [noise=true , address=45.91.116.5:4327 , piece_id=ULSWEIDH4LFO5WGT7JIPEL4F7JSQBLWJARZUL44OBNLUC6LOUGOQ, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:40.700510821 (502h7m40.571593856s) [noise=true , address=77.90.31.13:4653 , piece_id=4NSWDLVEHASULNLN4L2SD7YJEOK2HGGSABSZKULLX6NVRACVF6DQ, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:40.794940292 (502h7m40.666023327s) [noise=true , address=77.90.31.10:4532 , piece_id=4KEC7MSAKLWFQ5LG27CWZHZWC4WYUOBVPSLHKZVVDO2HB2V72NIQ, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:40.825821145 (502h7m40.69690419s) [noise=true , address=77.90.31.12:4595 , piece_id=SGG2LCPF3VDF5JAF5PC7RREY35SR5F5AIWOACTLWSQZXLZUMDZKA, seg_pos={0 0}, upload=440669]: Done uploading piece. err: <nil> Dec 26 13:36:40.825846833 (502h7m40.696929869s) [seg_pos={0 0}, upload=440669]: Segment reached optimal threshold of 14 pieces. Dec 26 13:36:40.826053557 (502h7m40.697136592s) [noise=true , address=77.90.31.21:4893 , piece_id=JJKUTOFJLGEALQT2ULTA2SOGJWCQ7RHA7NTTUD7VYQTI5BJS4C4Q, seg_pos={0 0}, upload=440669]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC): protocol: expected piece hash; context canceled; EOF Dec 26 13:36:40.826061944 (502h7m40.697144989s) [noise=true , address=77.90.31.21:4893 , piece_id=JJKUTOFJLGEALQT2ULTA2SOGJWCQ7RHA7NTTUD7VYQTI5BJS4C4Q, seg_pos={0 0}, upload=440669]: Long tail cancellation due to optimal threshold Dec 26 13:36:40.826104385 (502h7m40.69718743s) [noise=true , address=64.49.13.169:4716 , piece_id=4YKHMLTI2MIJCADDUM7CMLDPQLJO6BKMXQULZ6RF5HKQ22YW5WJA, seg_pos={0 0}, upload=440669]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:49264->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:49264->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49264->64.49.13.169:4716: use of closed network connection Dec 26 13:36:40.826120344 (502h7m40.69720338s) [noise=true , address=64.49.13.169:4716 , piece_id=4YKHMLTI2MIJCADDUM7CMLDPQLJO6BKMXQULZ6RF5HKQ22YW5WJA, seg_pos={0 0}, upload=440669]: Long tail cancellation due to optimal threshold Dec 26 13:36:40.826190167 (502h7m40.697273202s) [noise=true , address=45.91.116.4:4963 , piece_id=YCD5GJFEZMXAGAPZBYNM7SUR3ZDKM3ASKHMS5NEMRDZEDDF6DTGA, seg_pos={0 0}, upload=440669]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; context canceled; EOF Dec 26 13:36:40.826196339 (502h7m40.697279384s) [noise=true , address=45.91.116.4:4963 , piece_id=YCD5GJFEZMXAGAPZBYNM7SUR3ZDKM3ASKHMS5NEMRDZEDDF6DTGA, seg_pos={0 0}, upload=440669]: Long tail cancellation due to optimal threshold Dec 26 13:36:40.826273687 (502h7m40.697356722s) [noise=true , address=45.91.116.12:4253 , piece_id=VOOVGVFPIXWOTCNAZHMVQAEBNAVOGSYDI7GN5W3H3EVW3UJTQ3ZA, seg_pos={0 0}, upload=440669]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gDvXYhShpWvptTU8bivGGwYBTyMF3PJcRv6RJ66vm8aZewejn): protocol: expected piece hash; context canceled; EOF Dec 26 13:36:40.826279708 (502h7m40.697362743s) [noise=true , address=45.91.116.12:4253 , piece_id=VOOVGVFPIXWOTCNAZHMVQAEBNAVOGSYDI7GN5W3H3EVW3UJTQ3ZA, seg_pos={0 0}, upload=440669]: Long tail cancellation due to optimal threshold Dec 26 13:36:40.826308654 (502h7m40.697391689s) [seg_pos={0 0}, upload=440669]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 13:36:40.826430335 (502h7m40.69751337s) [upload=440669]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:36:40.826435234 (502h7m40.697518279s) [upload=440669]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:36:41.158299667 (502h7m41.029382702s) [upload=440669]: Done uploading segments... Dec 26 13:36:41.158324554 (502h7m41.0294076s) [upload=440669]: Upload finished. err: <nil> Duration: 1m0.002881905s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [..X..X....XXXXXX...XX.....X...XX.X.XXX.XXXXX..X.X.X.X...XXXX] duration: 5.001563415s trace: https://jaeger.dev.storj.io/trace/4b9683e993ef9f16 FAIL (2025-12-26 13:37:25 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001585568s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [...X..XXXXXX..X....X..X.X....X......XX.X.X......X.X.XX.X.X..] duration: 6.076804708s trace: https://jaeger.dev.storj.io/trace/194925e05da101b6 FAIL (2025-12-26 13:37:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: ULGNQ2N3KCU4AJ2KOG3MPB4ZA22A7PISAYEIONS22GYF3SJ7YVVA) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: V3ZLFYQWMBA7WCOWKRGT3BWKV4P24HUWLJ3QYU55E3QRSGOLNMPA) context deadline exceeded ======LOGS====== Dec 26 13:35:37.410141194 (502h6m37.281224239s) [upload=440658]: Starting upload Dec 26 13:35:37.410200077 (502h6m37.281283112s) [upload=440658]: Done starting upload Dec 26 13:35:37.410371974 (502h6m37.281455009s) [upload=440658]: Uploading segments... Dec 26 13:35:37.410405818 (502h6m37.281488854s) [upload=440658]: Got next segment. Inline: false Dec 26 13:35:37.410430295 (502h6m37.28151334s) [upload=440658]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 13:35:37.410434724 (502h6m37.281517759s) [upload=440658]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 13:35:37.658365881 (502h6m37.529448916s) [seg_pos={0 0}, upload=440658]: Begin upload segment... Dec 26 13:35:37.658692543 (502h6m37.529775588s) [seg_pos={0 0}, upload=440658]: Done begin upload segment. Dec 26 13:35:37.658734644 (502h6m37.529817689s) [noise=true , address=145.79.227.5:4222 , piece_id=XAH7PYSN4B73LHNMAZLS4PBSWLL7LVCI7TT6TMMNQLJYJ6BTJOFA, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:37.658837309 (502h6m37.529920354s) [noise=true , address=145.79.227.11:4729 , piece_id=UPM4TSGJJ6BTYREXCTFFXWU4DWGVY3SJGJRHTNT7GIMDYNSVQCQA, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:37.659430509 (502h6m37.530513544s) [noise=true , address=77.90.31.21:4880 , piece_id=V3ZLFYQWMBA7WCOWKRGT3BWKV4P24HUWLJ3QYU55E3QRSGOLNMPA, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:37.659929960 (502h6m37.531012995s) [noise=true , address=45.91.116.12:4227 , piece_id=DVNSRBA6RMNFYHA6SQCBJRCQQSIVTRZM37BDCZZ5KJPNE6GM4AYQ, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:37.660410465 (502h6m37.53149351s) [noise=true , address=145.79.227.13:4804 , piece_id=DH6VKCCJ4ZFOCXDNUNZZGVQWDLFT6UOXIOKP2HWWWFPSTTO6ZYOQ, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:37.660854030 (502h6m37.531937075s) [noise=true , address=145.79.227.28:4727 , piece_id=TKBT6GXOKTCIF63TGIZ7MOZT4KE2QEZWDFGUX36KGCAIILW346MQ, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:37.661268959 (502h6m37.532352004s) [noise=true , address=77.90.31.17:4773 , piece_id=3NNBTR4HJGW7FFGMYOOHDPFHI5PNMP4NMOAVI7XU5U2C55DEXKWA, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:37.661671285 (502h6m37.53275433s) [noise=true , address=145.79.227.3:4151 , piece_id=TPFUHIPNUTWST6SGBOC354COHBWCPT5CJQPY2X6LYXA3IES4IRMQ, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:37.661949365 (502h6m37.5330324s) [noise=true , address=145.79.227.29:4942 , piece_id=ANUMVLJ6RBE4LKNOIMOC7HXCN6U6F7DPQ3LEDNTHFIDIVZBIGOPA, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:37.662284774 (502h6m37.533367819s) [noise=true , address=145.79.227.45:4230 , piece_id=IAXLP3BPG7XXQVNG373A4HIRE4Y5QSQUGTFCAZI7KM3XAXO44L4A, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:37.662313618 (502h6m37.533396663s) [noise=true , address=145.79.227.43:4158 , piece_id=AX4QE4LDHC2JXTC4FHG33PTM2CZ5Z2E3SKO67EDK7IFPCX352B2Q, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:37.662589784 (502h6m37.533672819s) [noise=true , address=145.79.227.48:4303 , piece_id=XPI7GXVCXB5EZJ7AY7HZVHTWDRIZGPELG5FFD445GZCSMYMFCHUQ, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:37.662957434 (502h6m37.534040469s) [noise=true , address=45.91.116.6:4976 , piece_id=ULGNQ2N3KCU4AJ2KOG3MPB4ZA22A7PISAYEIONS22GYF3SJ7YVVA, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:37.663321357 (502h6m37.534404402s) [noise=true , address=145.79.227.14:4840 , piece_id=6M6UZ63TF2TW3PCK6FGZ77ZA2QTXI3FPH2HWHT3Z4KU77MTXNSSA, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:37.663819807 (502h6m37.534902852s) [noise=true , address=145.79.227.39:4015 , piece_id=YICLC5FARJWOS4KNI2LKW45W23IZDUVDZFKY7HXIWGD3E6XK2KFA, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:37.674164324 (502h6m37.545247369s) [noise=true , address=45.91.116.11:4117 , piece_id=2IRPKH6CD4CTYNUSNQ6HPCTQOBS5EQMWA6WYTUXTQROK6TRSPR2Q, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:39.381300663 (502h6m39.252383708s) [upload=440658]: Next returned nil segment Dec 26 13:35:39.381315812 (502h6m39.252398848s) [upload=440658]: Waiting for error group managing segments... Dec 26 13:35:39.664211929 (502h6m39.535294974s) [noise=true , address=145.79.227.11:4729 , piece_id=UPM4TSGJJ6BTYREXCTFFXWU4DWGVY3SJGJRHTNT7GIMDYNSVQCQA, seg_pos={0 0}, upload=440658]: Done uploading piece. err: <nil> Dec 26 13:35:39.665038062 (502h6m39.536121097s) [noise=true , address=145.79.227.13:4804 , piece_id=DH6VKCCJ4ZFOCXDNUNZZGVQWDLFT6UOXIOKP2HWWWFPSTTO6ZYOQ, seg_pos={0 0}, upload=440658]: Done uploading piece. err: <nil> Dec 26 13:35:39.677310511 (502h6m39.548393557s) [noise=true , address=145.79.227.43:4158 , piece_id=AX4QE4LDHC2JXTC4FHG33PTM2CZ5Z2E3SKO67EDK7IFPCX352B2Q, seg_pos={0 0}, upload=440658]: Done uploading piece. err: <nil> Dec 26 13:35:39.678086479 (502h6m39.549169514s) [noise=true , address=145.79.227.3:4151 , piece_id=TPFUHIPNUTWST6SGBOC354COHBWCPT5CJQPY2X6LYXA3IES4IRMQ, seg_pos={0 0}, upload=440658]: Done uploading piece. err: <nil> Dec 26 13:35:39.680371961 (502h6m39.551455006s) [noise=true , address=145.79.227.14:4840 , piece_id=6M6UZ63TF2TW3PCK6FGZ77ZA2QTXI3FPH2HWHT3Z4KU77MTXNSSA, seg_pos={0 0}, upload=440658]: Done uploading piece. err: <nil> Dec 26 13:35:39.680532376 (502h6m39.551615411s) [noise=true , address=145.79.227.48:4303 , piece_id=XPI7GXVCXB5EZJ7AY7HZVHTWDRIZGPELG5FFD445GZCSMYMFCHUQ, seg_pos={0 0}, upload=440658]: Done uploading piece. err: <nil> Dec 26 13:35:39.680940113 (502h6m39.552023158s) [noise=true , address=145.79.227.39:4015 , piece_id=YICLC5FARJWOS4KNI2LKW45W23IZDUVDZFKY7HXIWGD3E6XK2KFA, seg_pos={0 0}, upload=440658]: Done uploading piece. err: <nil> Dec 26 13:35:39.685290857 (502h6m39.556373902s) [noise=true , address=145.79.227.5:4222 , piece_id=XAH7PYSN4B73LHNMAZLS4PBSWLL7LVCI7TT6TMMNQLJYJ6BTJOFA, seg_pos={0 0}, upload=440658]: Done uploading piece. err: <nil> Dec 26 13:35:39.687983164 (502h6m39.5590662s) [noise=true , address=145.79.227.29:4942 , piece_id=ANUMVLJ6RBE4LKNOIMOC7HXCN6U6F7DPQ3LEDNTHFIDIVZBIGOPA, seg_pos={0 0}, upload=440658]: Done uploading piece. err: <nil> Dec 26 13:35:39.689775086 (502h6m39.560858121s) [noise=true , address=145.79.227.45:4230 , piece_id=IAXLP3BPG7XXQVNG373A4HIRE4Y5QSQUGTFCAZI7KM3XAXO44L4A, seg_pos={0 0}, upload=440658]: Done uploading piece. err: <nil> Dec 26 13:35:39.692473845 (502h6m39.563556891s) [noise=true , address=145.79.227.28:4727 , piece_id=TKBT6GXOKTCIF63TGIZ7MOZT4KE2QEZWDFGUX36KGCAIILW346MQ, seg_pos={0 0}, upload=440658]: Done uploading piece. err: <nil> Dec 26 13:35:39.896481834 (502h6m39.767564869s) [noise=true , address=77.90.31.21:4880 , piece_id=V3ZLFYQWMBA7WCOWKRGT3BWKV4P24HUWLJ3QYU55E3QRSGOLNMPA, seg_pos={0 0}, upload=440658]: Done uploading piece. err: <nil> Dec 26 13:35:39.906430739 (502h6m39.777513784s) [noise=true , address=45.91.116.6:4976 , piece_id=ULGNQ2N3KCU4AJ2KOG3MPB4ZA22A7PISAYEIONS22GYF3SJ7YVVA, seg_pos={0 0}, upload=440658]: Done uploading piece. err: <nil> Dec 26 13:35:39.906451259 (502h6m39.777534294s) [seg_pos={0 0}, upload=440658]: Segment reached optimal threshold of 13 pieces. Dec 26 13:35:39.906651539 (502h6m39.777734584s) [noise=true , address=77.90.31.17:4773 , piece_id=3NNBTR4HJGW7FFGMYOOHDPFHI5PNMP4NMOAVI7XU5U2C55DEXKWA, seg_pos={0 0}, upload=440658]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:39196->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:39196->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39196->77.90.31.17:4773: use of closed network connection Dec 26 13:35:39.906657421 (502h6m39.777740466s) [noise=true , address=77.90.31.17:4773 , piece_id=3NNBTR4HJGW7FFGMYOOHDPFHI5PNMP4NMOAVI7XU5U2C55DEXKWA, seg_pos={0 0}, upload=440658]: Long tail cancellation due to optimal threshold Dec 26 13:35:39.906874063 (502h6m39.777957109s) [noise=true , address=45.91.116.12:4227 , piece_id=DVNSRBA6RMNFYHA6SQCBJRCQQSIVTRZM37BDCZZ5KJPNE6GM4AYQ, seg_pos={0 0}, upload=440658]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 26 13:35:39.906891967 (502h6m39.777975002s) [noise=true , address=45.91.116.12:4227 , piece_id=DVNSRBA6RMNFYHA6SQCBJRCQQSIVTRZM37BDCZZ5KJPNE6GM4AYQ, seg_pos={0 0}, upload=440658]: Long tail cancellation due to optimal threshold Dec 26 13:35:39.906897248 (502h6m39.777980283s) [noise=true , address=45.91.116.11:4117 , piece_id=2IRPKH6CD4CTYNUSNQ6HPCTQOBS5EQMWA6WYTUXTQROK6TRSPR2Q, seg_pos={0 0}, upload=440658]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 26 13:35:39.906903420 (502h6m39.777986455s) [noise=true , address=45.91.116.11:4117 , piece_id=2IRPKH6CD4CTYNUSNQ6HPCTQOBS5EQMWA6WYTUXTQROK6TRSPR2Q, seg_pos={0 0}, upload=440658]: Long tail cancellation due to optimal threshold Dec 26 13:35:39.906927374 (502h6m39.77801042s) [seg_pos={0 0}, upload=440658]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 13:35:39.907017897 (502h6m39.778100942s) [upload=440658]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:35:39.907024399 (502h6m39.778107434s) [upload=440658]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:35:40.207390964 (502h6m40.078473999s) [upload=440658]: Done uploading segments... Dec 26 13:35:40.207438916 (502h6m40.078521961s) [upload=440658]: Upload finished. err: <nil> Duration: 6.07683157s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 18 out of 60 is failed (threshold=3) [.XXX...X..X...................XX.X........X.XXX.X.X..X.X.X.X] duration: 1m0.003217251s trace: https://jaeger.dev.storj.io/trace/5765fa49e8ce5bf9 FAIL (2025-12-26 13:36:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 31.19:4875 , piece_id=CYCK6J3LGXTI2MUJQCQDCDIK6E2MSGD44KO2J3FG7JGTZF4QSRWQ, seg_pos={0 0}, upload=440657]: Uploading piece... Dec 26 13:35:37.393118420 (502h6m37.264201455s) [noise=true , address=64.49.13.138:4481 , piece_id=4G5SLXORIGYQYYTUHTOOXCODYA6ZEUDRLYIH4A2VZOCCU2RMT4VA, seg_pos={0 0}, upload=440657]: Uploading piece... Dec 26 13:35:37.393380009 (502h6m37.264463044s) [noise=true , address=45.91.116.7:4028 , piece_id=6PUBEA4QVU6SWPERCB7HMARB2LVBETY7FIQWVVW6DHCPQNZTP5HA, seg_pos={0 0}, upload=440657]: Uploading piece... Dec 26 13:35:37.393676833 (502h6m37.264759868s) [noise=true , address=145.79.227.40:4075 , piece_id=DJJFNKCOD2ITGFD5PBX3I3H3JA2ECPX5L7H5P2TWV3RUUGEMH2SA, seg_pos={0 0}, upload=440657]: Uploading piece... Dec 26 13:35:37.394082356 (502h6m37.265165401s) [noise=true , address=145.79.227.43:4178 , piece_id=FLZHXMZBWHMZBMTIEIM5G6OK7HXWREZD5FHOUIY4BKBFSNSPF6RA, seg_pos={0 0}, upload=440657]: Uploading piece... Dec 26 13:35:37.394133403 (502h6m37.265216449s) [noise=true , address=145.79.227.44:4220 , piece_id=EVQL7GV5ULTXXVU3ZJ43BLHYZFDTKA7PSQCK5OVHBWBXKLJLB76A, seg_pos={0 0}, upload=440657]: Uploading piece... Dec 26 13:35:37.394387357 (502h6m37.265470402s) [noise=true , address=77.90.31.18:4821 , piece_id=W222PAYY6QCG3A6MXO22VLN4BCKAUSC5F33GTDI7RYS4U6OZI75Q, seg_pos={0 0}, upload=440657]: Uploading piece... Dec 26 13:35:37.394608217 (502h6m37.265691262s) [noise=true , address=77.90.31.12:4619 , piece_id=ZVRX55R67JZYMC6FW24YB5PASYJMKPBKBLEINFLXUE6AG343CJGA, seg_pos={0 0}, upload=440657]: Uploading piece... Dec 26 13:35:37.394654907 (502h6m37.265737952s) [noise=true , address=64.49.13.169:4710 , piece_id=P6O2TZZVD4DPOXPYISQE6TBBPVWS2VAIK26V2GKMIH62QNFIG4UA, seg_pos={0 0}, upload=440657]: Uploading piece... Dec 26 13:35:37.395038918 (502h6m37.266121973s) [noise=true , address=145.79.227.34:4926 , piece_id=AUKDZ6FDJPBXHLF4OKSYQ3NDS3PRTSYDBAVYBXQQETVC6HSJBJBQ, seg_pos={0 0}, upload=440657]: Uploading piece... Dec 26 13:35:37.395449188 (502h6m37.266532233s) [noise=true , address=77.90.31.17:4783 , piece_id=ELZSKFJX7YZFZV3XT4E4K2SCAGK6TULYNC64NHXO32OA7F7K2MPQ, seg_pos={0 0}, upload=440657]: Uploading piece... Dec 26 13:35:37.395873196 (502h6m37.266956241s) [noise=true , address=64.49.13.27:4960 , piece_id=W2D57N7A3YCXMAXVKO6C26A4UDGYM7RXZOQ64OMBGZK542MXQTSQ, seg_pos={0 0}, upload=440657]: Uploading piece... Dec 26 13:35:37.396348581 (502h6m37.267431626s) [noise=true , address=77.90.31.16:4746 , piece_id=3YHTJ7HCYOYNFZH7VSJHOT63BWN5HJSJGJYCNEACLGCEA7QVGIHQ, seg_pos={0 0}, upload=440657]: Uploading piece... Dec 26 13:35:37.396828765 (502h6m37.267911811s) [noise=true , address=77.90.31.10:4532 , piece_id=BH6T6KN6DWT3Q4IFQV3IOMHOFO5N2BEMFZQ3ZYEIBY46F45XE5RQ, seg_pos={0 0}, upload=440657]: Uploading piece... Dec 26 13:35:38.706921599 (502h6m38.578004644s) [upload=440657]: Next returned nil segment Dec 26 13:35:38.706936958 (502h6m38.578019993s) [upload=440657]: Waiting for error group managing segments... Dec 26 13:35:39.054316484 (502h6m38.925399519s) [noise=true , address=64.49.13.138:4481 , piece_id=4G5SLXORIGYQYYTUHTOOXCODYA6ZEUDRLYIH4A2VZOCCU2RMT4VA, seg_pos={0 0}, upload=440657]: Done uploading piece. err: <nil> Dec 26 13:35:39.265862367 (502h6m39.136945402s) [noise=true , address=145.79.227.43:4178 , piece_id=FLZHXMZBWHMZBMTIEIM5G6OK7HXWREZD5FHOUIY4BKBFSNSPF6RA, seg_pos={0 0}, upload=440657]: Done uploading piece. err: <nil> Dec 26 13:35:39.281121333 (502h6m39.152204368s) [noise=true , address=145.79.227.14:4855 , piece_id=X6SP4H4N4BPECVKJ6FHLD6MJXVCVS7TNNXDV3IZYNGUKDOJK5VAQ, seg_pos={0 0}, upload=440657]: Done uploading piece. err: <nil> Dec 26 13:35:39.396631707 (502h6m39.267714742s) [noise=true , address=64.49.13.27:4960 , piece_id=W2D57N7A3YCXMAXVKO6C26A4UDGYM7RXZOQ64OMBGZK542MXQTSQ, seg_pos={0 0}, upload=440657]: Done uploading piece. err: <nil> Dec 26 13:35:39.402218164 (502h6m39.273301199s) [noise=true , address=145.79.227.22:4516 , piece_id=P5UJQ3TR5Z3XDDDGI6AG3C3CD5FMO2U5VT6PG6JI2QPLF2XPNBAA, seg_pos={0 0}, upload=440657]: Done uploading piece. err: <nil> Dec 26 13:35:39.453948719 (502h6m39.325031834s) [noise=true , address=77.90.31.10:4532 , piece_id=BH6T6KN6DWT3Q4IFQV3IOMHOFO5N2BEMFZQ3ZYEIBY46F45XE5RQ, seg_pos={0 0}, upload=440657]: Done uploading piece. err: <nil> Dec 26 13:35:39.478747150 (502h6m39.349830195s) [noise=true , address=77.90.31.19:4875 , piece_id=CYCK6J3LGXTI2MUJQCQDCDIK6E2MSGD44KO2J3FG7JGTZF4QSRWQ, seg_pos={0 0}, upload=440657]: Done uploading piece. err: <nil> Dec 26 13:35:39.516106488 (502h6m39.387189523s) [noise=true , address=77.90.31.17:4783 , piece_id=ELZSKFJX7YZFZV3XT4E4K2SCAGK6TULYNC64NHXO32OA7F7K2MPQ, seg_pos={0 0}, upload=440657]: Done uploading piece. err: <nil> Dec 26 13:35:39.627098348 (502h6m39.498181383s) [noise=true , address=45.91.116.8:4050 , piece_id=CPBHDQI7Q27P4ZTJTDHLJCG7Q5476HAQK6XNXFJN326B3OOJCNWQ, seg_pos={0 0}, upload=440657]: Done uploading piece. err: <nil> Dec 26 13:35:40.106706702 (502h6m39.977789737s) [noise=true , address=77.90.31.18:4821 , piece_id=W222PAYY6QCG3A6MXO22VLN4BCKAUSC5F33GTDI7RYS4U6OZI75Q, seg_pos={0 0}, upload=440657]: Done uploading piece. err: <nil> Dec 26 13:35:40.202097544 (502h6m40.073180579s) [noise=true , address=145.79.227.44:4220 , piece_id=EVQL7GV5ULTXXVU3ZJ43BLHYZFDTKA7PSQCK5OVHBWBXKLJLB76A, seg_pos={0 0}, upload=440657]: Done uploading piece. err: <nil> Dec 26 13:35:40.241671461 (502h6m40.112755177s) [noise=true , address=145.79.227.30:4795 , piece_id=4M3IG4MGSSEK6FK5KXFSO7WF34QDQJA3X5DPDCDAJWENQX6OSQKA, seg_pos={0 0}, upload=440657]: Done uploading piece. err: <nil> Dec 26 13:35:40.249027247 (502h6m40.120110292s) [noise=true , address=145.79.227.40:4075 , piece_id=DJJFNKCOD2ITGFD5PBX3I3H3JA2ECPX5L7H5P2TWV3RUUGEMH2SA, seg_pos={0 0}, upload=440657]: Done uploading piece. err: <nil> Dec 26 13:35:40.271868671 (502h6m40.142951706s) [noise=true , address=145.79.227.34:4926 , piece_id=AUKDZ6FDJPBXHLF4OKSYQ3NDS3PRTSYDBAVYBXQQETVC6HSJBJBQ, seg_pos={0 0}, upload=440657]: Done uploading piece. err: <nil> Dec 26 13:35:40.271891675 (502h6m40.14297471s) [seg_pos={0 0}, upload=440657]: Segment reached optimal threshold of 14 pieces. Dec 26 13:35:40.272196795 (502h6m40.143279861s) [noise=true , address=64.49.13.169:4710 , piece_id=P6O2TZZVD4DPOXPYISQE6TBBPVWS2VAIK26V2GKMIH62QNFIG4UA, seg_pos={0 0}, upload=440657]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:45030->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:45030->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45030->64.49.13.169:4710: use of closed network connection Dec 26 13:35:40.272208909 (502h6m40.143291954s) [noise=true , address=64.49.13.169:4710 , piece_id=P6O2TZZVD4DPOXPYISQE6TBBPVWS2VAIK26V2GKMIH62QNFIG4UA, seg_pos={0 0}, upload=440657]: Long tail cancellation due to optimal threshold Dec 26 13:35:40.272232063 (502h6m40.143315098s) [noise=true , address=77.90.31.12:4619 , piece_id=ZVRX55R67JZYMC6FW24YB5PASYJMKPBKBLEINFLXUE6AG343CJGA, seg_pos={0 0}, upload=440657]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: write tcp 188.166.225.170:57946->77.90.31.12:4619: use of closed network connection; write tcp 188.166.225.170:57946->77.90.31.12:4619: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57946->77.90.31.12:4619: use of closed network connection Dec 26 13:35:40.272261228 (502h6m40.143344263s) [noise=true , address=77.90.31.12:4619 , piece_id=ZVRX55R67JZYMC6FW24YB5PASYJMKPBKBLEINFLXUE6AG343CJGA, seg_pos={0 0}, upload=440657]: Long tail cancellation due to optimal threshold Dec 26 13:35:40.272336291 (502h6m40.143419336s) [noise=true , address=77.90.31.16:4746 , piece_id=3YHTJ7HCYOYNFZH7VSJHOT63BWN5HJSJGJYCNEACLGCEA7QVGIHQ, seg_pos={0 0}, upload=440657]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o): protocol: write tcp 188.166.225.170:58514->77.90.31.16:4746: use of closed network connection; write tcp 188.166.225.170:58514->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58514->77.90.31.16:4746: use of closed network connection Dec 26 13:35:40.272342173 (502h6m40.143425208s) [noise=true , address=77.90.31.16:4746 , piece_id=3YHTJ7HCYOYNFZH7VSJHOT63BWN5HJSJGJYCNEACLGCEA7QVGIHQ, seg_pos={0 0}, upload=440657]: Long tail cancellation due to optimal threshold Dec 26 13:35:40.272517026 (502h6m40.143600071s) [noise=true , address=45.91.116.7:4028 , piece_id=6PUBEA4QVU6SWPERCB7HMARB2LVBETY7FIQWVVW6DHCPQNZTP5HA, seg_pos={0 0}, upload=440657]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g3sbXJir88dVbuJCShqFZ5KH2RJhq1kSqfVx6wN7ZkQtucnrB): protocol: expected piece hash; context canceled; EOF Dec 26 13:35:40.272523869 (502h6m40.143606914s) [noise=true , address=45.91.116.7:4028 , piece_id=6PUBEA4QVU6SWPERCB7HMARB2LVBETY7FIQWVVW6DHCPQNZTP5HA, seg_pos={0 0}, upload=440657]: Long tail cancellation due to optimal threshold Dec 26 13:35:40.272646833 (502h6m40.143729878s) [seg_pos={0 0}, upload=440657]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 13:35:40.272727647 (502h6m40.143810682s) [upload=440657]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:35:40.272734059 (502h6m40.143817114s) [upload=440657]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:35:40.568296787 (502h6m40.439379822s) [upload=440657]: Done uploading segments... Dec 26 13:35:40.568315101 (502h6m40.439398136s) [upload=440657]: Upload finished. err: <nil> Duration: 1m0.003227992s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [X..X..X....XXXXXX...XX.....X...XX.X.XXX.XXXXX..X.X.X.X...XXX] duration: 5.001278905s trace: https://jaeger.dev.storj.io/trace/20941fda93e203f FAIL (2025-12-26 13:36:33 UTC) orbiter warning: context deadline exceeded Duration: 5.001325824s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [....X..XXXXXX..X....X..X.X....X......XX.X.X......X.X.XX.X.X.] duration: 5.707048154s trace: https://jaeger.dev.storj.io/trace/194925e05da101b6 FAIL (2025-12-26 13:36:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: ULGNQ2N3KCU4AJ2KOG3MPB4ZA22A7PISAYEIONS22GYF3SJ7YVVA) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: V3ZLFYQWMBA7WCOWKRGT3BWKV4P24HUWLJ3QYU55E3QRSGOLNMPA) context deadline exceeded ======LOGS====== Dec 26 13:35:37.410141194 (502h6m37.281224239s) [upload=440658]: Starting upload Dec 26 13:35:37.410200077 (502h6m37.281283112s) [upload=440658]: Done starting upload Dec 26 13:35:37.410371974 (502h6m37.281455009s) [upload=440658]: Uploading segments... Dec 26 13:35:37.410405818 (502h6m37.281488854s) [upload=440658]: Got next segment. Inline: false Dec 26 13:35:37.410430295 (502h6m37.28151334s) [upload=440658]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 13:35:37.410434724 (502h6m37.281517759s) [upload=440658]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 13:35:37.658365881 (502h6m37.529448916s) [seg_pos={0 0}, upload=440658]: Begin upload segment... Dec 26 13:35:37.658692543 (502h6m37.529775588s) [seg_pos={0 0}, upload=440658]: Done begin upload segment. Dec 26 13:35:37.658734644 (502h6m37.529817689s) [noise=true , address=145.79.227.5:4222 , piece_id=XAH7PYSN4B73LHNMAZLS4PBSWLL7LVCI7TT6TMMNQLJYJ6BTJOFA, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:37.658837309 (502h6m37.529920354s) [noise=true , address=145.79.227.11:4729 , piece_id=UPM4TSGJJ6BTYREXCTFFXWU4DWGVY3SJGJRHTNT7GIMDYNSVQCQA, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:37.659430509 (502h6m37.530513544s) [noise=true , address=77.90.31.21:4880 , piece_id=V3ZLFYQWMBA7WCOWKRGT3BWKV4P24HUWLJ3QYU55E3QRSGOLNMPA, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:37.659929960 (502h6m37.531012995s) [noise=true , address=45.91.116.12:4227 , piece_id=DVNSRBA6RMNFYHA6SQCBJRCQQSIVTRZM37BDCZZ5KJPNE6GM4AYQ, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:37.660410465 (502h6m37.53149351s) [noise=true , address=145.79.227.13:4804 , piece_id=DH6VKCCJ4ZFOCXDNUNZZGVQWDLFT6UOXIOKP2HWWWFPSTTO6ZYOQ, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:37.660854030 (502h6m37.531937075s) [noise=true , address=145.79.227.28:4727 , piece_id=TKBT6GXOKTCIF63TGIZ7MOZT4KE2QEZWDFGUX36KGCAIILW346MQ, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:37.661268959 (502h6m37.532352004s) [noise=true , address=77.90.31.17:4773 , piece_id=3NNBTR4HJGW7FFGMYOOHDPFHI5PNMP4NMOAVI7XU5U2C55DEXKWA, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:37.661671285 (502h6m37.53275433s) [noise=true , address=145.79.227.3:4151 , piece_id=TPFUHIPNUTWST6SGBOC354COHBWCPT5CJQPY2X6LYXA3IES4IRMQ, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:37.661949365 (502h6m37.5330324s) [noise=true , address=145.79.227.29:4942 , piece_id=ANUMVLJ6RBE4LKNOIMOC7HXCN6U6F7DPQ3LEDNTHFIDIVZBIGOPA, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:37.662284774 (502h6m37.533367819s) [noise=true , address=145.79.227.45:4230 , piece_id=IAXLP3BPG7XXQVNG373A4HIRE4Y5QSQUGTFCAZI7KM3XAXO44L4A, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:37.662313618 (502h6m37.533396663s) [noise=true , address=145.79.227.43:4158 , piece_id=AX4QE4LDHC2JXTC4FHG33PTM2CZ5Z2E3SKO67EDK7IFPCX352B2Q, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:37.662589784 (502h6m37.533672819s) [noise=true , address=145.79.227.48:4303 , piece_id=XPI7GXVCXB5EZJ7AY7HZVHTWDRIZGPELG5FFD445GZCSMYMFCHUQ, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:37.662957434 (502h6m37.534040469s) [noise=true , address=45.91.116.6:4976 , piece_id=ULGNQ2N3KCU4AJ2KOG3MPB4ZA22A7PISAYEIONS22GYF3SJ7YVVA, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:37.663321357 (502h6m37.534404402s) [noise=true , address=145.79.227.14:4840 , piece_id=6M6UZ63TF2TW3PCK6FGZ77ZA2QTXI3FPH2HWHT3Z4KU77MTXNSSA, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:37.663819807 (502h6m37.534902852s) [noise=true , address=145.79.227.39:4015 , piece_id=YICLC5FARJWOS4KNI2LKW45W23IZDUVDZFKY7HXIWGD3E6XK2KFA, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:37.674164324 (502h6m37.545247369s) [noise=true , address=45.91.116.11:4117 , piece_id=2IRPKH6CD4CTYNUSNQ6HPCTQOBS5EQMWA6WYTUXTQROK6TRSPR2Q, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:39.381300663 (502h6m39.252383708s) [upload=440658]: Next returned nil segment Dec 26 13:35:39.381315812 (502h6m39.252398848s) [upload=440658]: Waiting for error group managing segments... Dec 26 13:35:39.664211929 (502h6m39.535294974s) [noise=true , address=145.79.227.11:4729 , piece_id=UPM4TSGJJ6BTYREXCTFFXWU4DWGVY3SJGJRHTNT7GIMDYNSVQCQA, seg_pos={0 0}, upload=440658]: Done uploading piece. err: <nil> Dec 26 13:35:39.665038062 (502h6m39.536121097s) [noise=true , address=145.79.227.13:4804 , piece_id=DH6VKCCJ4ZFOCXDNUNZZGVQWDLFT6UOXIOKP2HWWWFPSTTO6ZYOQ, seg_pos={0 0}, upload=440658]: Done uploading piece. err: <nil> Dec 26 13:35:39.677310511 (502h6m39.548393557s) [noise=true , address=145.79.227.43:4158 , piece_id=AX4QE4LDHC2JXTC4FHG33PTM2CZ5Z2E3SKO67EDK7IFPCX352B2Q, seg_pos={0 0}, upload=440658]: Done uploading piece. err: <nil> Dec 26 13:35:39.678086479 (502h6m39.549169514s) [noise=true , address=145.79.227.3:4151 , piece_id=TPFUHIPNUTWST6SGBOC354COHBWCPT5CJQPY2X6LYXA3IES4IRMQ, seg_pos={0 0}, upload=440658]: Done uploading piece. err: <nil> Dec 26 13:35:39.680371961 (502h6m39.551455006s) [noise=true , address=145.79.227.14:4840 , piece_id=6M6UZ63TF2TW3PCK6FGZ77ZA2QTXI3FPH2HWHT3Z4KU77MTXNSSA, seg_pos={0 0}, upload=440658]: Done uploading piece. err: <nil> Dec 26 13:35:39.680532376 (502h6m39.551615411s) [noise=true , address=145.79.227.48:4303 , piece_id=XPI7GXVCXB5EZJ7AY7HZVHTWDRIZGPELG5FFD445GZCSMYMFCHUQ, seg_pos={0 0}, upload=440658]: Done uploading piece. err: <nil> Dec 26 13:35:39.680940113 (502h6m39.552023158s) [noise=true , address=145.79.227.39:4015 , piece_id=YICLC5FARJWOS4KNI2LKW45W23IZDUVDZFKY7HXIWGD3E6XK2KFA, seg_pos={0 0}, upload=440658]: Done uploading piece. err: <nil> Dec 26 13:35:39.685290857 (502h6m39.556373902s) [noise=true , address=145.79.227.5:4222 , piece_id=XAH7PYSN4B73LHNMAZLS4PBSWLL7LVCI7TT6TMMNQLJYJ6BTJOFA, seg_pos={0 0}, upload=440658]: Done uploading piece. err: <nil> Dec 26 13:35:39.687983164 (502h6m39.5590662s) [noise=true , address=145.79.227.29:4942 , piece_id=ANUMVLJ6RBE4LKNOIMOC7HXCN6U6F7DPQ3LEDNTHFIDIVZBIGOPA, seg_pos={0 0}, upload=440658]: Done uploading piece. err: <nil> Dec 26 13:35:39.689775086 (502h6m39.560858121s) [noise=true , address=145.79.227.45:4230 , piece_id=IAXLP3BPG7XXQVNG373A4HIRE4Y5QSQUGTFCAZI7KM3XAXO44L4A, seg_pos={0 0}, upload=440658]: Done uploading piece. err: <nil> Dec 26 13:35:39.692473845 (502h6m39.563556891s) [noise=true , address=145.79.227.28:4727 , piece_id=TKBT6GXOKTCIF63TGIZ7MOZT4KE2QEZWDFGUX36KGCAIILW346MQ, seg_pos={0 0}, upload=440658]: Done uploading piece. err: <nil> Dec 26 13:35:39.896481834 (502h6m39.767564869s) [noise=true , address=77.90.31.21:4880 , piece_id=V3ZLFYQWMBA7WCOWKRGT3BWKV4P24HUWLJ3QYU55E3QRSGOLNMPA, seg_pos={0 0}, upload=440658]: Done uploading piece. err: <nil> Dec 26 13:35:39.906430739 (502h6m39.777513784s) [noise=true , address=45.91.116.6:4976 , piece_id=ULGNQ2N3KCU4AJ2KOG3MPB4ZA22A7PISAYEIONS22GYF3SJ7YVVA, seg_pos={0 0}, upload=440658]: Done uploading piece. err: <nil> Dec 26 13:35:39.906451259 (502h6m39.777534294s) [seg_pos={0 0}, upload=440658]: Segment reached optimal threshold of 13 pieces. Dec 26 13:35:39.906651539 (502h6m39.777734584s) [noise=true , address=77.90.31.17:4773 , piece_id=3NNBTR4HJGW7FFGMYOOHDPFHI5PNMP4NMOAVI7XU5U2C55DEXKWA, seg_pos={0 0}, upload=440658]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:39196->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:39196->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39196->77.90.31.17:4773: use of closed network connection Dec 26 13:35:39.906657421 (502h6m39.777740466s) [noise=true , address=77.90.31.17:4773 , piece_id=3NNBTR4HJGW7FFGMYOOHDPFHI5PNMP4NMOAVI7XU5U2C55DEXKWA, seg_pos={0 0}, upload=440658]: Long tail cancellation due to optimal threshold Dec 26 13:35:39.906874063 (502h6m39.777957109s) [noise=true , address=45.91.116.12:4227 , piece_id=DVNSRBA6RMNFYHA6SQCBJRCQQSIVTRZM37BDCZZ5KJPNE6GM4AYQ, seg_pos={0 0}, upload=440658]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 26 13:35:39.906891967 (502h6m39.777975002s) [noise=true , address=45.91.116.12:4227 , piece_id=DVNSRBA6RMNFYHA6SQCBJRCQQSIVTRZM37BDCZZ5KJPNE6GM4AYQ, seg_pos={0 0}, upload=440658]: Long tail cancellation due to optimal threshold Dec 26 13:35:39.906897248 (502h6m39.777980283s) [noise=true , address=45.91.116.11:4117 , piece_id=2IRPKH6CD4CTYNUSNQ6HPCTQOBS5EQMWA6WYTUXTQROK6TRSPR2Q, seg_pos={0 0}, upload=440658]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 26 13:35:39.906903420 (502h6m39.777986455s) [noise=true , address=45.91.116.11:4117 , piece_id=2IRPKH6CD4CTYNUSNQ6HPCTQOBS5EQMWA6WYTUXTQROK6TRSPR2Q, seg_pos={0 0}, upload=440658]: Long tail cancellation due to optimal threshold Dec 26 13:35:39.906927374 (502h6m39.77801042s) [seg_pos={0 0}, upload=440658]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 13:35:39.907017897 (502h6m39.778100942s) [upload=440658]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:35:39.907024399 (502h6m39.778107434s) [upload=440658]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:35:40.207390964 (502h6m40.078473999s) [upload=440658]: Done uploading segments... Dec 26 13:35:40.207438916 (502h6m40.078521961s) [upload=440658]: Upload finished. err: <nil> Duration: 5.707074163s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [.X..X..X....XXXXXX...XX.....X...XX.X.XXX.XXXXX..X.X.X.X...XX] duration: 5.002210019s trace: https://jaeger.dev.storj.io/trace/4ea48963cc0ee9e1 FAIL (2025-12-26 13:36:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002237042s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.....X..XXXXXX..X....X..X.X....X......XX.X.X......X.X.XX.X.X] duration: 30.003698026s trace: https://jaeger.dev.storj.io/trace/194925e05da101b6 FAIL (2025-12-26 13:36:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: ULGNQ2N3KCU4AJ2KOG3MPB4ZA22A7PISAYEIONS22GYF3SJ7YVVA) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: V3ZLFYQWMBA7WCOWKRGT3BWKV4P24HUWLJ3QYU55E3QRSGOLNMPA) context deadline exceeded ======LOGS====== Dec 26 13:35:37.410141194 (502h6m37.281224239s) [upload=440658]: Starting upload Dec 26 13:35:37.410200077 (502h6m37.281283112s) [upload=440658]: Done starting upload Dec 26 13:35:37.410371974 (502h6m37.281455009s) [upload=440658]: Uploading segments... Dec 26 13:35:37.410405818 (502h6m37.281488854s) [upload=440658]: Got next segment. Inline: false Dec 26 13:35:37.410430295 (502h6m37.28151334s) [upload=440658]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 13:35:37.410434724 (502h6m37.281517759s) [upload=440658]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 13:35:37.658365881 (502h6m37.529448916s) [seg_pos={0 0}, upload=440658]: Begin upload segment... Dec 26 13:35:37.658692543 (502h6m37.529775588s) [seg_pos={0 0}, upload=440658]: Done begin upload segment. Dec 26 13:35:37.658734644 (502h6m37.529817689s) [noise=true , address=145.79.227.5:4222 , piece_id=XAH7PYSN4B73LHNMAZLS4PBSWLL7LVCI7TT6TMMNQLJYJ6BTJOFA, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:37.658837309 (502h6m37.529920354s) [noise=true , address=145.79.227.11:4729 , piece_id=UPM4TSGJJ6BTYREXCTFFXWU4DWGVY3SJGJRHTNT7GIMDYNSVQCQA, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:37.659430509 (502h6m37.530513544s) [noise=true , address=77.90.31.21:4880 , piece_id=V3ZLFYQWMBA7WCOWKRGT3BWKV4P24HUWLJ3QYU55E3QRSGOLNMPA, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:37.659929960 (502h6m37.531012995s) [noise=true , address=45.91.116.12:4227 , piece_id=DVNSRBA6RMNFYHA6SQCBJRCQQSIVTRZM37BDCZZ5KJPNE6GM4AYQ, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:37.660410465 (502h6m37.53149351s) [noise=true , address=145.79.227.13:4804 , piece_id=DH6VKCCJ4ZFOCXDNUNZZGVQWDLFT6UOXIOKP2HWWWFPSTTO6ZYOQ, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:37.660854030 (502h6m37.531937075s) [noise=true , address=145.79.227.28:4727 , piece_id=TKBT6GXOKTCIF63TGIZ7MOZT4KE2QEZWDFGUX36KGCAIILW346MQ, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:37.661268959 (502h6m37.532352004s) [noise=true , address=77.90.31.17:4773 , piece_id=3NNBTR4HJGW7FFGMYOOHDPFHI5PNMP4NMOAVI7XU5U2C55DEXKWA, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:37.661671285 (502h6m37.53275433s) [noise=true , address=145.79.227.3:4151 , piece_id=TPFUHIPNUTWST6SGBOC354COHBWCPT5CJQPY2X6LYXA3IES4IRMQ, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:37.661949365 (502h6m37.5330324s) [noise=true , address=145.79.227.29:4942 , piece_id=ANUMVLJ6RBE4LKNOIMOC7HXCN6U6F7DPQ3LEDNTHFIDIVZBIGOPA, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:37.662284774 (502h6m37.533367819s) [noise=true , address=145.79.227.45:4230 , piece_id=IAXLP3BPG7XXQVNG373A4HIRE4Y5QSQUGTFCAZI7KM3XAXO44L4A, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:37.662313618 (502h6m37.533396663s) [noise=true , address=145.79.227.43:4158 , piece_id=AX4QE4LDHC2JXTC4FHG33PTM2CZ5Z2E3SKO67EDK7IFPCX352B2Q, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:37.662589784 (502h6m37.533672819s) [noise=true , address=145.79.227.48:4303 , piece_id=XPI7GXVCXB5EZJ7AY7HZVHTWDRIZGPELG5FFD445GZCSMYMFCHUQ, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:37.662957434 (502h6m37.534040469s) [noise=true , address=45.91.116.6:4976 , piece_id=ULGNQ2N3KCU4AJ2KOG3MPB4ZA22A7PISAYEIONS22GYF3SJ7YVVA, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:37.663321357 (502h6m37.534404402s) [noise=true , address=145.79.227.14:4840 , piece_id=6M6UZ63TF2TW3PCK6FGZ77ZA2QTXI3FPH2HWHT3Z4KU77MTXNSSA, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:37.663819807 (502h6m37.534902852s) [noise=true , address=145.79.227.39:4015 , piece_id=YICLC5FARJWOS4KNI2LKW45W23IZDUVDZFKY7HXIWGD3E6XK2KFA, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:37.674164324 (502h6m37.545247369s) [noise=true , address=45.91.116.11:4117 , piece_id=2IRPKH6CD4CTYNUSNQ6HPCTQOBS5EQMWA6WYTUXTQROK6TRSPR2Q, seg_pos={0 0}, upload=440658]: Uploading piece... Dec 26 13:35:39.381300663 (502h6m39.252383708s) [upload=440658]: Next returned nil segment Dec 26 13:35:39.381315812 (502h6m39.252398848s) [upload=440658]: Waiting for error group managing segments... Dec 26 13:35:39.664211929 (502h6m39.535294974s) [noise=true , address=145.79.227.11:4729 , piece_id=UPM4TSGJJ6BTYREXCTFFXWU4DWGVY3SJGJRHTNT7GIMDYNSVQCQA, seg_pos={0 0}, upload=440658]: Done uploading piece. err: <nil> Dec 26 13:35:39.665038062 (502h6m39.536121097s) [noise=true , address=145.79.227.13:4804 , piece_id=DH6VKCCJ4ZFOCXDNUNZZGVQWDLFT6UOXIOKP2HWWWFPSTTO6ZYOQ, seg_pos={0 0}, upload=440658]: Done uploading piece. err: <nil> Dec 26 13:35:39.677310511 (502h6m39.548393557s) [noise=true , address=145.79.227.43:4158 , piece_id=AX4QE4LDHC2JXTC4FHG33PTM2CZ5Z2E3SKO67EDK7IFPCX352B2Q, seg_pos={0 0}, upload=440658]: Done uploading piece. err: <nil> Dec 26 13:35:39.678086479 (502h6m39.549169514s) [noise=true , address=145.79.227.3:4151 , piece_id=TPFUHIPNUTWST6SGBOC354COHBWCPT5CJQPY2X6LYXA3IES4IRMQ, seg_pos={0 0}, upload=440658]: Done uploading piece. err: <nil> Dec 26 13:35:39.680371961 (502h6m39.551455006s) [noise=true , address=145.79.227.14:4840 , piece_id=6M6UZ63TF2TW3PCK6FGZ77ZA2QTXI3FPH2HWHT3Z4KU77MTXNSSA, seg_pos={0 0}, upload=440658]: Done uploading piece. err: <nil> Dec 26 13:35:39.680532376 (502h6m39.551615411s) [noise=true , address=145.79.227.48:4303 , piece_id=XPI7GXVCXB5EZJ7AY7HZVHTWDRIZGPELG5FFD445GZCSMYMFCHUQ, seg_pos={0 0}, upload=440658]: Done uploading piece. err: <nil> Dec 26 13:35:39.680940113 (502h6m39.552023158s) [noise=true , address=145.79.227.39:4015 , piece_id=YICLC5FARJWOS4KNI2LKW45W23IZDUVDZFKY7HXIWGD3E6XK2KFA, seg_pos={0 0}, upload=440658]: Done uploading piece. err: <nil> Dec 26 13:35:39.685290857 (502h6m39.556373902s) [noise=true , address=145.79.227.5:4222 , piece_id=XAH7PYSN4B73LHNMAZLS4PBSWLL7LVCI7TT6TMMNQLJYJ6BTJOFA, seg_pos={0 0}, upload=440658]: Done uploading piece. err: <nil> Dec 26 13:35:39.687983164 (502h6m39.5590662s) [noise=true , address=145.79.227.29:4942 , piece_id=ANUMVLJ6RBE4LKNOIMOC7HXCN6U6F7DPQ3LEDNTHFIDIVZBIGOPA, seg_pos={0 0}, upload=440658]: Done uploading piece. err: <nil> Dec 26 13:35:39.689775086 (502h6m39.560858121s) [noise=true , address=145.79.227.45:4230 , piece_id=IAXLP3BPG7XXQVNG373A4HIRE4Y5QSQUGTFCAZI7KM3XAXO44L4A, seg_pos={0 0}, upload=440658]: Done uploading piece. err: <nil> Dec 26 13:35:39.692473845 (502h6m39.563556891s) [noise=true , address=145.79.227.28:4727 , piece_id=TKBT6GXOKTCIF63TGIZ7MOZT4KE2QEZWDFGUX36KGCAIILW346MQ, seg_pos={0 0}, upload=440658]: Done uploading piece. err: <nil> Dec 26 13:35:39.896481834 (502h6m39.767564869s) [noise=true , address=77.90.31.21:4880 , piece_id=V3ZLFYQWMBA7WCOWKRGT3BWKV4P24HUWLJ3QYU55E3QRSGOLNMPA, seg_pos={0 0}, upload=440658]: Done uploading piece. err: <nil> Dec 26 13:35:39.906430739 (502h6m39.777513784s) [noise=true , address=45.91.116.6:4976 , piece_id=ULGNQ2N3KCU4AJ2KOG3MPB4ZA22A7PISAYEIONS22GYF3SJ7YVVA, seg_pos={0 0}, upload=440658]: Done uploading piece. err: <nil> Dec 26 13:35:39.906451259 (502h6m39.777534294s) [seg_pos={0 0}, upload=440658]: Segment reached optimal threshold of 13 pieces. Dec 26 13:35:39.906651539 (502h6m39.777734584s) [noise=true , address=77.90.31.17:4773 , piece_id=3NNBTR4HJGW7FFGMYOOHDPFHI5PNMP4NMOAVI7XU5U2C55DEXKWA, seg_pos={0 0}, upload=440658]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:39196->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:39196->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39196->77.90.31.17:4773: use of closed network connection Dec 26 13:35:39.906657421 (502h6m39.777740466s) [noise=true , address=77.90.31.17:4773 , piece_id=3NNBTR4HJGW7FFGMYOOHDPFHI5PNMP4NMOAVI7XU5U2C55DEXKWA, seg_pos={0 0}, upload=440658]: Long tail cancellation due to optimal threshold Dec 26 13:35:39.906874063 (502h6m39.777957109s) [noise=true , address=45.91.116.12:4227 , piece_id=DVNSRBA6RMNFYHA6SQCBJRCQQSIVTRZM37BDCZZ5KJPNE6GM4AYQ, seg_pos={0 0}, upload=440658]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 26 13:35:39.906891967 (502h6m39.777975002s) [noise=true , address=45.91.116.12:4227 , piece_id=DVNSRBA6RMNFYHA6SQCBJRCQQSIVTRZM37BDCZZ5KJPNE6GM4AYQ, seg_pos={0 0}, upload=440658]: Long tail cancellation due to optimal threshold Dec 26 13:35:39.906897248 (502h6m39.777980283s) [noise=true , address=45.91.116.11:4117 , piece_id=2IRPKH6CD4CTYNUSNQ6HPCTQOBS5EQMWA6WYTUXTQROK6TRSPR2Q, seg_pos={0 0}, upload=440658]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 26 13:35:39.906903420 (502h6m39.777986455s) [noise=true , address=45.91.116.11:4117 , piece_id=2IRPKH6CD4CTYNUSNQ6HPCTQOBS5EQMWA6WYTUXTQROK6TRSPR2Q, seg_pos={0 0}, upload=440658]: Long tail cancellation due to optimal threshold Dec 26 13:35:39.906927374 (502h6m39.77801042s) [seg_pos={0 0}, upload=440658]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 13:35:39.907017897 (502h6m39.778100942s) [upload=440658]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:35:39.907024399 (502h6m39.778107434s) [upload=440658]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:35:40.207390964 (502h6m40.078473999s) [upload=440658]: Done uploading segments... Dec 26 13:35:40.207438916 (502h6m40.078521961s) [upload=440658]: Upload finished. err: <nil> Duration: 30.003722333s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 18 out of 60 is failed (threshold=3) [X.XXX...X..X...................XX.X........X.XXX.X.X..X.X.X.] duration: 55.5776003s trace: https://jaeger.dev.storj.io/trace/3fb593cbe520bd05 FAIL (2025-12-26 13:35:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, Piece ID: UCXOYRVZHJXD3WTO4HDK6EI4XDNC5BJEIUTMQ7UMXD3XIDSMAVWQ) context deadline exceeded; piecestore: (Node ID: 12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, Piece ID: YTJJ5X7Y537C5EHWDMVXRFPDO5ALTR5JIE2M7EVT3HLAB4H7CQGA) context deadline exceeded; piecestore: (Node ID: 1k29MGpez3ByMHnG8qGBm9iPwo3VrKUGDHpz9CqGEGNn47rxDx, Piece ID: NODI4AIAELCQUE2EM24YOMZPOO7HVEHNQZ7QPJQXX64HZRYMLP4A) context deadline exceeded ======LOGS====== 23107s) [noise=true , address=145.79.227.44:4210 , piece_id=YYDUMRRH7SFDBGZVMCBCLHF65QMWF7TFGKZLSKN3ZNDPXOJQSNJA, seg_pos={0 0}, upload=440630]: Uploading piece... Dec 26 13:33:41.406480859 (502h4m41.277563894s) [noise=true , address=77.90.31.19:4848 , piece_id=52G73GXC4OODJLNJHWTWLIREZI5O364Y7ZMXSA2WXA57ALM3M3XA, seg_pos={0 0}, upload=440630]: Uploading piece... Dec 26 13:33:41.406575268 (502h4m41.277658313s) [noise=true , address=77.90.31.18:4816 , piece_id=UCXOYRVZHJXD3WTO4HDK6EI4XDNC5BJEIUTMQ7UMXD3XIDSMAVWQ, seg_pos={0 0}, upload=440630]: Uploading piece... Dec 26 13:33:41.406745132 (502h4m41.277828177s) [noise=true , address=77.90.31.11:4572 , piece_id=EGDASBRSA5ZKKWRYLFCBRB5SZQRKJ65CYTLKWAJE7IOZAVUOPICQ, seg_pos={0 0}, upload=440630]: Uploading piece... Dec 26 13:33:41.406949461 (502h4m41.278033077s) [noise=true , address=77.90.31.14:4679 , piece_id=NODI4AIAELCQUE2EM24YOMZPOO7HVEHNQZ7QPJQXX64HZRYMLP4A, seg_pos={0 0}, upload=440630]: Uploading piece... Dec 26 13:33:41.407005047 (502h4m41.278088082s) [noise=true , address=45.91.116.5:4313 , piece_id=Y5D2ZQN2RPA3UEVWYXSXNIESYSTTGNNFU3RHT7LM7PG4FDBU6OMQ, seg_pos={0 0}, upload=440630]: Uploading piece... Dec 26 13:33:41.407293316 (502h4m41.278376361s) [noise=true , address=145.79.227.40:4053 , piece_id=R6HUVGBWIZX7XB5L5UWZEDNOI6EQXOS2HAOG4UXMAN7UEYAH6GIQ, seg_pos={0 0}, upload=440630]: Uploading piece... Dec 26 13:33:41.407348862 (502h4m41.278431897s) [noise=true , address=77.90.31.17:4769 , piece_id=YTJJ5X7Y537C5EHWDMVXRFPDO5ALTR5JIE2M7EVT3HLAB4H7CQGA, seg_pos={0 0}, upload=440630]: Uploading piece... Dec 26 13:33:41.407543993 (502h4m41.278627028s) [noise=true , address=145.79.227.30:4786 , piece_id=DJHTV4YTHNKTW3LY6D3NKVMDBH7E4I5IGITYDUTS25XCGRIPP4OA, seg_pos={0 0}, upload=440630]: Uploading piece... Dec 26 13:33:41.407720880 (502h4m41.278803925s) [noise=true , address=45.91.116.11:4130 , piece_id=67MQFUMW6SPTIZAGSBTRYSZFSAY5OA7YWQC4SGYV6KTIRUGAP6YA, seg_pos={0 0}, upload=440630]: Uploading piece... Dec 26 13:33:41.407811432 (502h4m41.278894468s) [noise=true , address=77.90.31.13:4638 , piece_id=TYLH3WHGROKFVJDKBD7YE4TV7SYDZNOWTVAQUA32HFABEGNCY57Q, seg_pos={0 0}, upload=440630]: Uploading piece... Dec 26 13:33:41.408083020 (502h4m41.279166065s) [noise=true , address=77.90.31.15:4729 , piece_id=EG7N7UWUQ6VIF7HHWKHKPNHBRP5JLDT6MYTF27TBVULB4UFDDDLQ, seg_pos={0 0}, upload=440630]: Uploading piece... Dec 26 13:33:41.408134427 (502h4m41.279217462s) [noise=true , address=64.49.13.169:4710 , piece_id=AKQFG2CMMF5TW3HJXWHL4QCGKKD55VW4VZONCCD5BVNXKVLNT6HA, seg_pos={0 0}, upload=440630]: Uploading piece... Dec 26 13:33:41.408479264 (502h4m41.279562299s) [noise=true , address=145.79.227.17:4002 , piece_id=ENWLCA27UJZ27QZOXUW7CJ5FDJV54TFUIWMOKLK7HDP2BTM6TYJA, seg_pos={0 0}, upload=440630]: Uploading piece... Dec 26 13:33:41.408958787 (502h4m41.280041832s) [noise=true , address=145.79.227.2:4122 , piece_id=YX4ZCATYJNBZAKJPPMMKK5NLHYWKDX27ZPYSJZ575A4D5TYONLNA, seg_pos={0 0}, upload=440630]: Uploading piece... Dec 26 13:33:42.886221623 (502h4m42.757304668s) [upload=440630]: Next returned nil segment Dec 26 13:33:42.886237353 (502h4m42.757320398s) [upload=440630]: Waiting for error group managing segments... Dec 26 13:33:43.290181168 (502h4m43.161264213s) [noise=true , address=145.79.227.17:4002 , piece_id=ENWLCA27UJZ27QZOXUW7CJ5FDJV54TFUIWMOKLK7HDP2BTM6TYJA, seg_pos={0 0}, upload=440630]: Done uploading piece. err: <nil> Dec 26 13:33:43.299594993 (502h4m43.170678038s) [noise=true , address=145.79.227.27:4708 , piece_id=M6765TMVYXJXOD2H6EGT7QQYDVQ6L54VUQVBGISKQHGWLPV5F5VQ, seg_pos={0 0}, upload=440630]: Done uploading piece. err: <nil> Dec 26 13:33:43.303927914 (502h4m43.175010959s) [noise=true , address=145.79.227.2:4122 , piece_id=YX4ZCATYJNBZAKJPPMMKK5NLHYWKDX27ZPYSJZ575A4D5TYONLNA, seg_pos={0 0}, upload=440630]: Done uploading piece. err: <nil> Dec 26 13:33:43.470369601 (502h4m43.341452636s) [noise=true , address=77.90.31.14:4679 , piece_id=NODI4AIAELCQUE2EM24YOMZPOO7HVEHNQZ7QPJQXX64HZRYMLP4A, seg_pos={0 0}, upload=440630]: Done uploading piece. err: <nil> Dec 26 13:33:43.564479173 (502h4m43.435562218s) [noise=true , address=45.91.116.11:4130 , piece_id=67MQFUMW6SPTIZAGSBTRYSZFSAY5OA7YWQC4SGYV6KTIRUGAP6YA, seg_pos={0 0}, upload=440630]: Done uploading piece. err: <nil> Dec 26 13:33:43.634048146 (502h4m43.505131191s) [noise=true , address=45.91.116.5:4313 , piece_id=Y5D2ZQN2RPA3UEVWYXSXNIESYSTTGNNFU3RHT7LM7PG4FDBU6OMQ, seg_pos={0 0}, upload=440630]: Done uploading piece. err: <nil> Dec 26 13:33:43.645510602 (502h4m43.516593647s) [noise=true , address=77.90.31.18:4816 , piece_id=UCXOYRVZHJXD3WTO4HDK6EI4XDNC5BJEIUTMQ7UMXD3XIDSMAVWQ, seg_pos={0 0}, upload=440630]: Done uploading piece. err: <nil> Dec 26 13:33:43.660701748 (502h4m43.531784783s) [noise=true , address=45.91.116.6:4995 , piece_id=TA2C5D6WCNHAZC66JJKDZAFECQ7L5RNPPMIXXG7VW5KHJ2UWSDQA, seg_pos={0 0}, upload=440630]: Done uploading piece. err: <nil> Dec 26 13:33:44.136510990 (502h4m44.007594035s) [noise=true , address=77.90.31.13:4638 , piece_id=TYLH3WHGROKFVJDKBD7YE4TV7SYDZNOWTVAQUA32HFABEGNCY57Q, seg_pos={0 0}, upload=440630]: Done uploading piece. err: <nil> Dec 26 13:33:44.212561443 (502h4m44.083644478s) [noise=true , address=145.79.227.30:4786 , piece_id=DJHTV4YTHNKTW3LY6D3NKVMDBH7E4I5IGITYDUTS25XCGRIPP4OA, seg_pos={0 0}, upload=440630]: Done uploading piece. err: <nil> Dec 26 13:33:44.215809659 (502h4m44.086892694s) [noise=true , address=77.90.31.17:4769 , piece_id=YTJJ5X7Y537C5EHWDMVXRFPDO5ALTR5JIE2M7EVT3HLAB4H7CQGA, seg_pos={0 0}, upload=440630]: Done uploading piece. err: <nil> Dec 26 13:33:44.242389070 (502h4m44.113472115s) [noise=true , address=145.79.227.44:4210 , piece_id=YYDUMRRH7SFDBGZVMCBCLHF65QMWF7TFGKZLSKN3ZNDPXOJQSNJA, seg_pos={0 0}, upload=440630]: Done uploading piece. err: <nil> Dec 26 13:33:44.344702805 (502h4m44.21578584s) [noise=true , address=145.79.227.40:4053 , piece_id=R6HUVGBWIZX7XB5L5UWZEDNOI6EQXOS2HAOG4UXMAN7UEYAH6GIQ, seg_pos={0 0}, upload=440630]: Done uploading piece. err: <nil> Dec 26 13:33:44.603185995 (502h4m44.47426903s) [noise=true , address=77.90.31.15:4729 , piece_id=EG7N7UWUQ6VIF7HHWKHKPNHBRP5JLDT6MYTF27TBVULB4UFDDDLQ, seg_pos={0 0}, upload=440630]: Done uploading piece. err: <nil> Dec 26 13:33:44.603204641 (502h4m44.474287676s) [seg_pos={0 0}, upload=440630]: Segment reached optimal threshold of 14 pieces. Dec 26 13:33:44.603386137 (502h4m44.474469172s) [noise=true , address=77.90.31.19:4848 , piece_id=52G73GXC4OODJLNJHWTWLIREZI5O364Y7ZMXSA2WXA57ALM3M3XA, seg_pos={0 0}, upload=440630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD): protocol: write tcp 188.166.225.170:46878->77.90.31.19:4848: use of closed network connection; write tcp 188.166.225.170:46878->77.90.31.19:4848: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46878->77.90.31.19:4848: use of closed network connection Dec 26 13:33:44.603402048 (502h4m44.474485083s) [noise=true , address=77.90.31.19:4848 , piece_id=52G73GXC4OODJLNJHWTWLIREZI5O364Y7ZMXSA2WXA57ALM3M3XA, seg_pos={0 0}, upload=440630]: Long tail cancellation due to optimal threshold Dec 26 13:33:44.603730072 (502h4m44.474813117s) [noise=true , address=64.49.13.169:4710 , piece_id=AKQFG2CMMF5TW3HJXWHL4QCGKKD55VW4VZONCCD5BVNXKVLNT6HA, seg_pos={0 0}, upload=440630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:45650->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:45650->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45650->64.49.13.169:4710: use of closed network connection Dec 26 13:33:44.603740231 (502h4m44.474823266s) [noise=true , address=64.49.13.169:4710 , piece_id=AKQFG2CMMF5TW3HJXWHL4QCGKKD55VW4VZONCCD5BVNXKVLNT6HA, seg_pos={0 0}, upload=440630]: Long tail cancellation due to optimal threshold Dec 26 13:33:44.603908942 (502h4m44.474991987s) [noise=true , address=145.79.227.11:4749 , piece_id=DKJJB7KRMZERWFK2NALFNNEXFQ2WI4RM674DFCCBOMTDFIFUL3BQ, seg_pos={0 0}, upload=440630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sRotj46WjNNsVciJDRvSEbrj8XHwshKRmMneedu6UH4egmZDm): protocol: expected piece hash; context canceled; EOF Dec 26 13:33:44.603915164 (502h4m44.474998209s) [noise=true , address=145.79.227.11:4749 , piece_id=DKJJB7KRMZERWFK2NALFNNEXFQ2WI4RM674DFCCBOMTDFIFUL3BQ, seg_pos={0 0}, upload=440630]: Long tail cancellation due to optimal threshold Dec 26 13:33:44.604069178 (502h4m44.475152213s) [noise=true , address=77.90.31.11:4572 , piece_id=EGDASBRSA5ZKKWRYLFCBRB5SZQRKJ65CYTLKWAJE7IOZAVUOPICQ, seg_pos={0 0}, upload=440630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7): protocol: expected piece hash; context canceled; EOF Dec 26 13:33:44.604075259 (502h4m44.475158294s) [noise=true , address=77.90.31.11:4572 , piece_id=EGDASBRSA5ZKKWRYLFCBRB5SZQRKJ65CYTLKWAJE7IOZAVUOPICQ, seg_pos={0 0}, upload=440630]: Long tail cancellation due to optimal threshold Dec 26 13:33:44.604107100 (502h4m44.475190135s) [seg_pos={0 0}, upload=440630]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 13:33:44.604179508 (502h4m44.475262543s) [upload=440630]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:33:44.604184307 (502h4m44.475267342s) [upload=440630]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:33:44.906902750 (502h4m44.777985785s) [upload=440630]: Done uploading segments... Dec 26 13:33:44.906932617 (502h4m44.778015662s) [upload=440630]: Upload finished. err: <nil> Duration: 55.577622191s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [X.X..X..X....XXXXXX...XX.....X...XX.X.XXX.XXXXX..X.X.X.X...X] duration: 5.001275336s trace: https://jaeger.dev.storj.io/trace/47a99afc66181d2f FAIL (2025-12-26 13:35:23 UTC) orbiter warning: context deadline exceeded Duration: 5.001299511s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X.....X..XXXXXX..X....X..X.X....X......XX.X.X......X.X.XX.X.] duration: 6.772880156s trace: https://jaeger.dev.storj.io/trace/5ab4c6da8a68398c FAIL (2025-12-26 13:34:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: FOUXBR6DGALTKP3HTMRCILLXAE7NP2YOTPM6F4P6NDB2YKIJZMAQ) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: FZ3K4D3EBFTQQCWE6Y63EQUOERCAYSSY6UPPI4JMDVBT55DPLHKQ) context deadline exceeded ======LOGS====== load Dec 26 13:34:14.778817077 (502h5m14.649900122s) [upload=440635]: Uploading segments... Dec 26 13:34:14.779038018 (502h5m14.650121053s) [upload=440635]: Got next segment. Inline: false Dec 26 13:34:14.779066201 (502h5m14.650149236s) [upload=440635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 13:34:14.779070630 (502h5m14.650153665s) [upload=440635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 13:34:15.024799484 (502h5m14.895882539s) [seg_pos={0 0}, upload=440635]: Begin upload segment... Dec 26 13:34:15.024856413 (502h5m14.895939448s) [seg_pos={0 0}, upload=440635]: Done begin upload segment. Dec 26 13:34:15.024886901 (502h5m14.895969956s) [noise=true , address=145.79.227.45:4231 , piece_id=T57HHFFHHZNRKXWI7EZMBLNU5Y73VTRMNB7I4YROF7SAEYCRFZAA, seg_pos={0 0}, upload=440635]: Uploading piece... Dec 26 13:34:15.024895757 (502h5m14.895978792s) [noise=true , address=45.91.116.7:4009 , piece_id=FZ3K4D3EBFTQQCWE6Y63EQUOERCAYSSY6UPPI4JMDVBT55DPLHKQ, seg_pos={0 0}, upload=440635]: Uploading piece... Dec 26 13:34:15.024970640 (502h5m14.896053685s) [noise=true , address=145.79.227.48:4302 , piece_id=OAD2S346NJSHA7VAJ7AZ6ABTEHTPBDXI5XRGYGV2U7D4BK425PAA, seg_pos={0 0}, upload=440635]: Uploading piece... Dec 26 13:34:15.025576273 (502h5m14.896659318s) [noise=true , address=145.79.227.16:4911 , piece_id=KJWN7FKBVHZKJ2KSRP35UKXYWHN3KKLACHIYFVW44CK2ZTBBZX7A, seg_pos={0 0}, upload=440635]: Uploading piece... Dec 26 13:34:15.026139055 (502h5m14.8972221s) [noise=true , address=145.79.227.43:4157 , piece_id=SWU32W5EL54WGN4VYKDRECCCTRP45GYLZYJ4VISFZPUORKMCFR7A, seg_pos={0 0}, upload=440635]: Uploading piece... Dec 26 13:34:15.026707668 (502h5m14.897790723s) [noise=true , address=145.79.227.19:4077 , piece_id=X5TSI2MBU5UBMIDS6J3UGQUIOA2KJ3QD3JX6LRBIW6LLDXIKH3YA, seg_pos={0 0}, upload=440635]: Uploading piece... Dec 26 13:34:15.027179395 (502h5m14.89826244s) [noise=true , address=145.79.227.11:4729 , piece_id=TPUQLFW4UR3F3VC5KLFY4N2J4PQVBUJYEQEXCB76ZT3HOI5YVG4Q, seg_pos={0 0}, upload=440635]: Uploading piece... Dec 26 13:34:15.028294098 (502h5m14.899377133s) [noise=true , address=145.79.227.38:4979 , piece_id=TG6SB3UEKN4NTJBINK6L5DKKL7UACH5C4BSVEECRP53WE5OJCFNA, seg_pos={0 0}, upload=440635]: Uploading piece... Dec 26 13:34:15.028581957 (502h5m14.899664992s) [noise=true , address=145.79.227.49:4337 , piece_id=TBFBK5SGJ327KO4ATYEJZOOYL6ZG63H3KZ2YABXCUTLCAK5UJ4TQ, seg_pos={0 0}, upload=440635]: Uploading piece... Dec 26 13:34:15.030317652 (502h5m14.901400697s) [noise=true , address=145.79.227.17:4970 , piece_id=ZZT27MMGHGS6TNPPH6EQJW4J6F66MW7CIFOGBZUS2SNESKTRXJ6A, seg_pos={0 0}, upload=440635]: Uploading piece... Dec 26 13:34:15.031805164 (502h5m14.902888219s) [noise=true , address=145.79.227.35:4481 , piece_id=DLUACD5E4FDVAEWANC6UQO22LRIEEBUXY4IRA5BAH7MQMFQ4ADFQ, seg_pos={0 0}, upload=440635]: Uploading piece... Dec 26 13:34:15.033104418 (502h5m14.904187463s) [noise=true , address=45.91.116.9:4262 , piece_id=FOUXBR6DGALTKP3HTMRCILLXAE7NP2YOTPM6F4P6NDB2YKIJZMAQ, seg_pos={0 0}, upload=440635]: Uploading piece... Dec 26 13:34:15.034155239 (502h5m14.905238284s) [noise=true , address=45.91.116.13:4155 , piece_id=DF6ZE5VAIWUOTAKJDSADATX4DUK3QZSPPSUR3ITSTHU4O5RVWRLQ, seg_pos={0 0}, upload=440635]: Uploading piece... Dec 26 13:34:15.034635684 (502h5m14.905718729s) [noise=true , address=145.79.227.4:4185 , piece_id=2E6X72JRAZZY7K46V2Y66YE6TJZGNPLCIZ5IVWAU5GW32YGG7E5Q, seg_pos={0 0}, upload=440635]: Uploading piece... Dec 26 13:34:15.035910481 (502h5m14.906993526s) [noise=true , address=77.90.31.13:4629 , piece_id=I3XTI5SOJRFJVFMNHQLQPAXQ3PVI4YXTJDY4VAM2Z5HYQRCNK2FQ, seg_pos={0 0}, upload=440635]: Uploading piece... Dec 26 13:34:15.036585547 (502h5m14.907668592s) [noise=true , address=45.91.116.6:4974 , piece_id=HP2WEPGZEI25WIFGQEQUAXN7VR4ZN3VKFBLFI757FAHFFTGU2OMQ, seg_pos={0 0}, upload=440635]: Uploading piece... Dec 26 13:34:16.644690800 (502h5m16.515773835s) [upload=440635]: Next returned nil segment Dec 26 13:34:16.644704577 (502h5m16.515787622s) [upload=440635]: Waiting for error group managing segments... Dec 26 13:34:17.032053813 (502h5m16.903136858s) [noise=true , address=145.79.227.11:4729 , piece_id=TPUQLFW4UR3F3VC5KLFY4N2J4PQVBUJYEQEXCB76ZT3HOI5YVG4Q, seg_pos={0 0}, upload=440635]: Done uploading piece. err: <nil> Dec 26 13:34:17.033907272 (502h5m16.904990317s) [noise=true , address=145.79.227.43:4157 , piece_id=SWU32W5EL54WGN4VYKDRECCCTRP45GYLZYJ4VISFZPUORKMCFR7A, seg_pos={0 0}, upload=440635]: Done uploading piece. err: <nil> Dec 26 13:34:17.038210747 (502h5m16.909293782s) [noise=true , address=145.79.227.19:4077 , piece_id=X5TSI2MBU5UBMIDS6J3UGQUIOA2KJ3QD3JX6LRBIW6LLDXIKH3YA, seg_pos={0 0}, upload=440635]: Done uploading piece. err: <nil> Dec 26 13:34:17.042918561 (502h5m16.914001606s) [noise=true , address=145.79.227.17:4970 , piece_id=ZZT27MMGHGS6TNPPH6EQJW4J6F66MW7CIFOGBZUS2SNESKTRXJ6A, seg_pos={0 0}, upload=440635]: Done uploading piece. err: <nil> Dec 26 13:34:17.043990322 (502h5m16.915073367s) [noise=true , address=145.79.227.35:4481 , piece_id=DLUACD5E4FDVAEWANC6UQO22LRIEEBUXY4IRA5BAH7MQMFQ4ADFQ, seg_pos={0 0}, upload=440635]: Done uploading piece. err: <nil> Dec 26 13:34:17.044524068 (502h5m16.915607113s) [noise=true , address=145.79.227.38:4979 , piece_id=TG6SB3UEKN4NTJBINK6L5DKKL7UACH5C4BSVEECRP53WE5OJCFNA, seg_pos={0 0}, upload=440635]: Done uploading piece. err: <nil> Dec 26 13:34:17.047476049 (502h5m16.918559094s) [noise=true , address=145.79.227.16:4911 , piece_id=KJWN7FKBVHZKJ2KSRP35UKXYWHN3KKLACHIYFVW44CK2ZTBBZX7A, seg_pos={0 0}, upload=440635]: Done uploading piece. err: <nil> Dec 26 13:34:17.060620359 (502h5m16.931703404s) [noise=true , address=145.79.227.48:4302 , piece_id=OAD2S346NJSHA7VAJ7AZ6ABTEHTPBDXI5XRGYGV2U7D4BK425PAA, seg_pos={0 0}, upload=440635]: Done uploading piece. err: <nil> Dec 26 13:34:17.066393322 (502h5m16.937476367s) [noise=true , address=145.79.227.4:4185 , piece_id=2E6X72JRAZZY7K46V2Y66YE6TJZGNPLCIZ5IVWAU5GW32YGG7E5Q, seg_pos={0 0}, upload=440635]: Done uploading piece. err: <nil> Dec 26 13:34:17.068333586 (502h5m16.939416621s) [noise=true , address=145.79.227.45:4231 , piece_id=T57HHFFHHZNRKXWI7EZMBLNU5Y73VTRMNB7I4YROF7SAEYCRFZAA, seg_pos={0 0}, upload=440635]: Done uploading piece. err: <nil> Dec 26 13:34:17.074562688 (502h5m16.945645733s) [noise=true , address=145.79.227.49:4337 , piece_id=TBFBK5SGJ327KO4ATYEJZOOYL6ZG63H3KZ2YABXCUTLCAK5UJ4TQ, seg_pos={0 0}, upload=440635]: Done uploading piece. err: <nil> Dec 26 13:34:17.310446826 (502h5m17.181529861s) [noise=true , address=45.91.116.9:4262 , piece_id=FOUXBR6DGALTKP3HTMRCILLXAE7NP2YOTPM6F4P6NDB2YKIJZMAQ, seg_pos={0 0}, upload=440635]: Done uploading piece. err: <nil> Dec 26 13:34:17.348265650 (502h5m17.219348695s) [noise=true , address=45.91.116.7:4009 , piece_id=FZ3K4D3EBFTQQCWE6Y63EQUOERCAYSSY6UPPI4JMDVBT55DPLHKQ, seg_pos={0 0}, upload=440635]: Done uploading piece. err: <nil> Dec 26 13:34:17.348292129 (502h5m17.219375175s) [seg_pos={0 0}, upload=440635]: Segment reached optimal threshold of 13 pieces. Dec 26 13:34:17.348630895 (502h5m17.21971395s) [noise=true , address=45.91.116.6:4974 , piece_id=HP2WEPGZEI25WIFGQEQUAXN7VR4ZN3VKFBLFI757FAHFFTGU2OMQ, seg_pos={0 0}, upload=440635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:35228->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:35228->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35228->45.91.116.6:4974: use of closed network connection Dec 26 13:34:17.348641174 (502h5m17.219724209s) [noise=true , address=45.91.116.6:4974 , piece_id=HP2WEPGZEI25WIFGQEQUAXN7VR4ZN3VKFBLFI757FAHFFTGU2OMQ, seg_pos={0 0}, upload=440635]: Long tail cancellation due to optimal threshold Dec 26 13:34:17.348881893 (502h5m17.219964938s) [noise=true , address=45.91.116.13:4155 , piece_id=DF6ZE5VAIWUOTAKJDSADATX4DUK3QZSPPSUR3ITSTHU4O5RVWRLQ, seg_pos={0 0}, upload=440635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: read tcp 188.166.225.170:57816->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 13:34:17.348902703 (502h5m17.219985758s) [noise=true , address=45.91.116.13:4155 , piece_id=DF6ZE5VAIWUOTAKJDSADATX4DUK3QZSPPSUR3ITSTHU4O5RVWRLQ, seg_pos={0 0}, upload=440635]: Long tail cancellation due to optimal threshold Dec 26 13:34:17.348958189 (502h5m17.220041284s) [noise=true , address=77.90.31.13:4629 , piece_id=I3XTI5SOJRFJVFMNHQLQPAXQ3PVI4YXTJDY4VAM2Z5HYQRCNK2FQ, seg_pos={0 0}, upload=440635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: read tcp 188.166.225.170:53800->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 13:34:17.348968969 (502h5m17.220052004s) [noise=true , address=77.90.31.13:4629 , piece_id=I3XTI5SOJRFJVFMNHQLQPAXQ3PVI4YXTJDY4VAM2Z5HYQRCNK2FQ, seg_pos={0 0}, upload=440635]: Long tail cancellation due to optimal threshold Dec 26 13:34:17.348987123 (502h5m17.220070158s) [seg_pos={0 0}, upload=440635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 13:34:17.349106160 (502h5m17.220189205s) [upload=440635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:34:17.349113804 (502h5m17.22019685s) [upload=440635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:34:17.635851607 (502h5m17.506934642s) [upload=440635]: Done uploading segments... Dec 26 13:34:17.635870924 (502h5m17.506953959s) [upload=440635]: Upload finished. err: <nil> Duration: 6.773016726s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XX.....X..XXXXXX..X....X..X.X....X......XX.X.X......X.X.XX.X] duration: 30.00353933s trace: https://jaeger.dev.storj.io/trace/5ab4c6da8a68398c FAIL (2025-12-26 13:34:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: FOUXBR6DGALTKP3HTMRCILLXAE7NP2YOTPM6F4P6NDB2YKIJZMAQ) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: FZ3K4D3EBFTQQCWE6Y63EQUOERCAYSSY6UPPI4JMDVBT55DPLHKQ) context deadline exceeded ======LOGS====== load Dec 26 13:34:14.778817077 (502h5m14.649900122s) [upload=440635]: Uploading segments... Dec 26 13:34:14.779038018 (502h5m14.650121053s) [upload=440635]: Got next segment. Inline: false Dec 26 13:34:14.779066201 (502h5m14.650149236s) [upload=440635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 13:34:14.779070630 (502h5m14.650153665s) [upload=440635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 13:34:15.024799484 (502h5m14.895882539s) [seg_pos={0 0}, upload=440635]: Begin upload segment... Dec 26 13:34:15.024856413 (502h5m14.895939448s) [seg_pos={0 0}, upload=440635]: Done begin upload segment. Dec 26 13:34:15.024886901 (502h5m14.895969956s) [noise=true , address=145.79.227.45:4231 , piece_id=T57HHFFHHZNRKXWI7EZMBLNU5Y73VTRMNB7I4YROF7SAEYCRFZAA, seg_pos={0 0}, upload=440635]: Uploading piece... Dec 26 13:34:15.024895757 (502h5m14.895978792s) [noise=true , address=45.91.116.7:4009 , piece_id=FZ3K4D3EBFTQQCWE6Y63EQUOERCAYSSY6UPPI4JMDVBT55DPLHKQ, seg_pos={0 0}, upload=440635]: Uploading piece... Dec 26 13:34:15.024970640 (502h5m14.896053685s) [noise=true , address=145.79.227.48:4302 , piece_id=OAD2S346NJSHA7VAJ7AZ6ABTEHTPBDXI5XRGYGV2U7D4BK425PAA, seg_pos={0 0}, upload=440635]: Uploading piece... Dec 26 13:34:15.025576273 (502h5m14.896659318s) [noise=true , address=145.79.227.16:4911 , piece_id=KJWN7FKBVHZKJ2KSRP35UKXYWHN3KKLACHIYFVW44CK2ZTBBZX7A, seg_pos={0 0}, upload=440635]: Uploading piece... Dec 26 13:34:15.026139055 (502h5m14.8972221s) [noise=true , address=145.79.227.43:4157 , piece_id=SWU32W5EL54WGN4VYKDRECCCTRP45GYLZYJ4VISFZPUORKMCFR7A, seg_pos={0 0}, upload=440635]: Uploading piece... Dec 26 13:34:15.026707668 (502h5m14.897790723s) [noise=true , address=145.79.227.19:4077 , piece_id=X5TSI2MBU5UBMIDS6J3UGQUIOA2KJ3QD3JX6LRBIW6LLDXIKH3YA, seg_pos={0 0}, upload=440635]: Uploading piece... Dec 26 13:34:15.027179395 (502h5m14.89826244s) [noise=true , address=145.79.227.11:4729 , piece_id=TPUQLFW4UR3F3VC5KLFY4N2J4PQVBUJYEQEXCB76ZT3HOI5YVG4Q, seg_pos={0 0}, upload=440635]: Uploading piece... Dec 26 13:34:15.028294098 (502h5m14.899377133s) [noise=true , address=145.79.227.38:4979 , piece_id=TG6SB3UEKN4NTJBINK6L5DKKL7UACH5C4BSVEECRP53WE5OJCFNA, seg_pos={0 0}, upload=440635]: Uploading piece... Dec 26 13:34:15.028581957 (502h5m14.899664992s) [noise=true , address=145.79.227.49:4337 , piece_id=TBFBK5SGJ327KO4ATYEJZOOYL6ZG63H3KZ2YABXCUTLCAK5UJ4TQ, seg_pos={0 0}, upload=440635]: Uploading piece... Dec 26 13:34:15.030317652 (502h5m14.901400697s) [noise=true , address=145.79.227.17:4970 , piece_id=ZZT27MMGHGS6TNPPH6EQJW4J6F66MW7CIFOGBZUS2SNESKTRXJ6A, seg_pos={0 0}, upload=440635]: Uploading piece... Dec 26 13:34:15.031805164 (502h5m14.902888219s) [noise=true , address=145.79.227.35:4481 , piece_id=DLUACD5E4FDVAEWANC6UQO22LRIEEBUXY4IRA5BAH7MQMFQ4ADFQ, seg_pos={0 0}, upload=440635]: Uploading piece... Dec 26 13:34:15.033104418 (502h5m14.904187463s) [noise=true , address=45.91.116.9:4262 , piece_id=FOUXBR6DGALTKP3HTMRCILLXAE7NP2YOTPM6F4P6NDB2YKIJZMAQ, seg_pos={0 0}, upload=440635]: Uploading piece... Dec 26 13:34:15.034155239 (502h5m14.905238284s) [noise=true , address=45.91.116.13:4155 , piece_id=DF6ZE5VAIWUOTAKJDSADATX4DUK3QZSPPSUR3ITSTHU4O5RVWRLQ, seg_pos={0 0}, upload=440635]: Uploading piece... Dec 26 13:34:15.034635684 (502h5m14.905718729s) [noise=true , address=145.79.227.4:4185 , piece_id=2E6X72JRAZZY7K46V2Y66YE6TJZGNPLCIZ5IVWAU5GW32YGG7E5Q, seg_pos={0 0}, upload=440635]: Uploading piece... Dec 26 13:34:15.035910481 (502h5m14.906993526s) [noise=true , address=77.90.31.13:4629 , piece_id=I3XTI5SOJRFJVFMNHQLQPAXQ3PVI4YXTJDY4VAM2Z5HYQRCNK2FQ, seg_pos={0 0}, upload=440635]: Uploading piece... Dec 26 13:34:15.036585547 (502h5m14.907668592s) [noise=true , address=45.91.116.6:4974 , piece_id=HP2WEPGZEI25WIFGQEQUAXN7VR4ZN3VKFBLFI757FAHFFTGU2OMQ, seg_pos={0 0}, upload=440635]: Uploading piece... Dec 26 13:34:16.644690800 (502h5m16.515773835s) [upload=440635]: Next returned nil segment Dec 26 13:34:16.644704577 (502h5m16.515787622s) [upload=440635]: Waiting for error group managing segments... Dec 26 13:34:17.032053813 (502h5m16.903136858s) [noise=true , address=145.79.227.11:4729 , piece_id=TPUQLFW4UR3F3VC5KLFY4N2J4PQVBUJYEQEXCB76ZT3HOI5YVG4Q, seg_pos={0 0}, upload=440635]: Done uploading piece. err: <nil> Dec 26 13:34:17.033907272 (502h5m16.904990317s) [noise=true , address=145.79.227.43:4157 , piece_id=SWU32W5EL54WGN4VYKDRECCCTRP45GYLZYJ4VISFZPUORKMCFR7A, seg_pos={0 0}, upload=440635]: Done uploading piece. err: <nil> Dec 26 13:34:17.038210747 (502h5m16.909293782s) [noise=true , address=145.79.227.19:4077 , piece_id=X5TSI2MBU5UBMIDS6J3UGQUIOA2KJ3QD3JX6LRBIW6LLDXIKH3YA, seg_pos={0 0}, upload=440635]: Done uploading piece. err: <nil> Dec 26 13:34:17.042918561 (502h5m16.914001606s) [noise=true , address=145.79.227.17:4970 , piece_id=ZZT27MMGHGS6TNPPH6EQJW4J6F66MW7CIFOGBZUS2SNESKTRXJ6A, seg_pos={0 0}, upload=440635]: Done uploading piece. err: <nil> Dec 26 13:34:17.043990322 (502h5m16.915073367s) [noise=true , address=145.79.227.35:4481 , piece_id=DLUACD5E4FDVAEWANC6UQO22LRIEEBUXY4IRA5BAH7MQMFQ4ADFQ, seg_pos={0 0}, upload=440635]: Done uploading piece. err: <nil> Dec 26 13:34:17.044524068 (502h5m16.915607113s) [noise=true , address=145.79.227.38:4979 , piece_id=TG6SB3UEKN4NTJBINK6L5DKKL7UACH5C4BSVEECRP53WE5OJCFNA, seg_pos={0 0}, upload=440635]: Done uploading piece. err: <nil> Dec 26 13:34:17.047476049 (502h5m16.918559094s) [noise=true , address=145.79.227.16:4911 , piece_id=KJWN7FKBVHZKJ2KSRP35UKXYWHN3KKLACHIYFVW44CK2ZTBBZX7A, seg_pos={0 0}, upload=440635]: Done uploading piece. err: <nil> Dec 26 13:34:17.060620359 (502h5m16.931703404s) [noise=true , address=145.79.227.48:4302 , piece_id=OAD2S346NJSHA7VAJ7AZ6ABTEHTPBDXI5XRGYGV2U7D4BK425PAA, seg_pos={0 0}, upload=440635]: Done uploading piece. err: <nil> Dec 26 13:34:17.066393322 (502h5m16.937476367s) [noise=true , address=145.79.227.4:4185 , piece_id=2E6X72JRAZZY7K46V2Y66YE6TJZGNPLCIZ5IVWAU5GW32YGG7E5Q, seg_pos={0 0}, upload=440635]: Done uploading piece. err: <nil> Dec 26 13:34:17.068333586 (502h5m16.939416621s) [noise=true , address=145.79.227.45:4231 , piece_id=T57HHFFHHZNRKXWI7EZMBLNU5Y73VTRMNB7I4YROF7SAEYCRFZAA, seg_pos={0 0}, upload=440635]: Done uploading piece. err: <nil> Dec 26 13:34:17.074562688 (502h5m16.945645733s) [noise=true , address=145.79.227.49:4337 , piece_id=TBFBK5SGJ327KO4ATYEJZOOYL6ZG63H3KZ2YABXCUTLCAK5UJ4TQ, seg_pos={0 0}, upload=440635]: Done uploading piece. err: <nil> Dec 26 13:34:17.310446826 (502h5m17.181529861s) [noise=true , address=45.91.116.9:4262 , piece_id=FOUXBR6DGALTKP3HTMRCILLXAE7NP2YOTPM6F4P6NDB2YKIJZMAQ, seg_pos={0 0}, upload=440635]: Done uploading piece. err: <nil> Dec 26 13:34:17.348265650 (502h5m17.219348695s) [noise=true , address=45.91.116.7:4009 , piece_id=FZ3K4D3EBFTQQCWE6Y63EQUOERCAYSSY6UPPI4JMDVBT55DPLHKQ, seg_pos={0 0}, upload=440635]: Done uploading piece. err: <nil> Dec 26 13:34:17.348292129 (502h5m17.219375175s) [seg_pos={0 0}, upload=440635]: Segment reached optimal threshold of 13 pieces. Dec 26 13:34:17.348630895 (502h5m17.21971395s) [noise=true , address=45.91.116.6:4974 , piece_id=HP2WEPGZEI25WIFGQEQUAXN7VR4ZN3VKFBLFI757FAHFFTGU2OMQ, seg_pos={0 0}, upload=440635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:35228->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:35228->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35228->45.91.116.6:4974: use of closed network connection Dec 26 13:34:17.348641174 (502h5m17.219724209s) [noise=true , address=45.91.116.6:4974 , piece_id=HP2WEPGZEI25WIFGQEQUAXN7VR4ZN3VKFBLFI757FAHFFTGU2OMQ, seg_pos={0 0}, upload=440635]: Long tail cancellation due to optimal threshold Dec 26 13:34:17.348881893 (502h5m17.219964938s) [noise=true , address=45.91.116.13:4155 , piece_id=DF6ZE5VAIWUOTAKJDSADATX4DUK3QZSPPSUR3ITSTHU4O5RVWRLQ, seg_pos={0 0}, upload=440635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: read tcp 188.166.225.170:57816->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 13:34:17.348902703 (502h5m17.219985758s) [noise=true , address=45.91.116.13:4155 , piece_id=DF6ZE5VAIWUOTAKJDSADATX4DUK3QZSPPSUR3ITSTHU4O5RVWRLQ, seg_pos={0 0}, upload=440635]: Long tail cancellation due to optimal threshold Dec 26 13:34:17.348958189 (502h5m17.220041284s) [noise=true , address=77.90.31.13:4629 , piece_id=I3XTI5SOJRFJVFMNHQLQPAXQ3PVI4YXTJDY4VAM2Z5HYQRCNK2FQ, seg_pos={0 0}, upload=440635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: read tcp 188.166.225.170:53800->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 13:34:17.348968969 (502h5m17.220052004s) [noise=true , address=77.90.31.13:4629 , piece_id=I3XTI5SOJRFJVFMNHQLQPAXQ3PVI4YXTJDY4VAM2Z5HYQRCNK2FQ, seg_pos={0 0}, upload=440635]: Long tail cancellation due to optimal threshold Dec 26 13:34:17.348987123 (502h5m17.220070158s) [seg_pos={0 0}, upload=440635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 13:34:17.349106160 (502h5m17.220189205s) [upload=440635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:34:17.349113804 (502h5m17.22019685s) [upload=440635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:34:17.635851607 (502h5m17.506934642s) [upload=440635]: Done uploading segments... Dec 26 13:34:17.635870924 (502h5m17.506953959s) [upload=440635]: Upload finished. err: <nil> Duration: 30.003557765s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X.XXX...X..X...................XX.X........X.XXX.X.X..X.X.X] duration: 1m0.002736309s trace: https://jaeger.dev.storj.io/trace/3fb593cbe520bd05 FAIL (2025-12-26 13:34:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, Piece ID: UCXOYRVZHJXD3WTO4HDK6EI4XDNC5BJEIUTMQ7UMXD3XIDSMAVWQ) context deadline exceeded; piecestore: (Node ID: 12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, Piece ID: YTJJ5X7Y537C5EHWDMVXRFPDO5ALTR5JIE2M7EVT3HLAB4H7CQGA) context deadline exceeded; piecestore: (Node ID: 1k29MGpez3ByMHnG8qGBm9iPwo3VrKUGDHpz9CqGEGNn47rxDx, Piece ID: NODI4AIAELCQUE2EM24YOMZPOO7HVEHNQZ7QPJQXX64HZRYMLP4A) context deadline exceeded ======LOGS====== 23107s) [noise=true , address=145.79.227.44:4210 , piece_id=YYDUMRRH7SFDBGZVMCBCLHF65QMWF7TFGKZLSKN3ZNDPXOJQSNJA, seg_pos={0 0}, upload=440630]: Uploading piece... Dec 26 13:33:41.406480859 (502h4m41.277563894s) [noise=true , address=77.90.31.19:4848 , piece_id=52G73GXC4OODJLNJHWTWLIREZI5O364Y7ZMXSA2WXA57ALM3M3XA, seg_pos={0 0}, upload=440630]: Uploading piece... Dec 26 13:33:41.406575268 (502h4m41.277658313s) [noise=true , address=77.90.31.18:4816 , piece_id=UCXOYRVZHJXD3WTO4HDK6EI4XDNC5BJEIUTMQ7UMXD3XIDSMAVWQ, seg_pos={0 0}, upload=440630]: Uploading piece... Dec 26 13:33:41.406745132 (502h4m41.277828177s) [noise=true , address=77.90.31.11:4572 , piece_id=EGDASBRSA5ZKKWRYLFCBRB5SZQRKJ65CYTLKWAJE7IOZAVUOPICQ, seg_pos={0 0}, upload=440630]: Uploading piece... Dec 26 13:33:41.406949461 (502h4m41.278033077s) [noise=true , address=77.90.31.14:4679 , piece_id=NODI4AIAELCQUE2EM24YOMZPOO7HVEHNQZ7QPJQXX64HZRYMLP4A, seg_pos={0 0}, upload=440630]: Uploading piece... Dec 26 13:33:41.407005047 (502h4m41.278088082s) [noise=true , address=45.91.116.5:4313 , piece_id=Y5D2ZQN2RPA3UEVWYXSXNIESYSTTGNNFU3RHT7LM7PG4FDBU6OMQ, seg_pos={0 0}, upload=440630]: Uploading piece... Dec 26 13:33:41.407293316 (502h4m41.278376361s) [noise=true , address=145.79.227.40:4053 , piece_id=R6HUVGBWIZX7XB5L5UWZEDNOI6EQXOS2HAOG4UXMAN7UEYAH6GIQ, seg_pos={0 0}, upload=440630]: Uploading piece... Dec 26 13:33:41.407348862 (502h4m41.278431897s) [noise=true , address=77.90.31.17:4769 , piece_id=YTJJ5X7Y537C5EHWDMVXRFPDO5ALTR5JIE2M7EVT3HLAB4H7CQGA, seg_pos={0 0}, upload=440630]: Uploading piece... Dec 26 13:33:41.407543993 (502h4m41.278627028s) [noise=true , address=145.79.227.30:4786 , piece_id=DJHTV4YTHNKTW3LY6D3NKVMDBH7E4I5IGITYDUTS25XCGRIPP4OA, seg_pos={0 0}, upload=440630]: Uploading piece... Dec 26 13:33:41.407720880 (502h4m41.278803925s) [noise=true , address=45.91.116.11:4130 , piece_id=67MQFUMW6SPTIZAGSBTRYSZFSAY5OA7YWQC4SGYV6KTIRUGAP6YA, seg_pos={0 0}, upload=440630]: Uploading piece... Dec 26 13:33:41.407811432 (502h4m41.278894468s) [noise=true , address=77.90.31.13:4638 , piece_id=TYLH3WHGROKFVJDKBD7YE4TV7SYDZNOWTVAQUA32HFABEGNCY57Q, seg_pos={0 0}, upload=440630]: Uploading piece... Dec 26 13:33:41.408083020 (502h4m41.279166065s) [noise=true , address=77.90.31.15:4729 , piece_id=EG7N7UWUQ6VIF7HHWKHKPNHBRP5JLDT6MYTF27TBVULB4UFDDDLQ, seg_pos={0 0}, upload=440630]: Uploading piece... Dec 26 13:33:41.408134427 (502h4m41.279217462s) [noise=true , address=64.49.13.169:4710 , piece_id=AKQFG2CMMF5TW3HJXWHL4QCGKKD55VW4VZONCCD5BVNXKVLNT6HA, seg_pos={0 0}, upload=440630]: Uploading piece... Dec 26 13:33:41.408479264 (502h4m41.279562299s) [noise=true , address=145.79.227.17:4002 , piece_id=ENWLCA27UJZ27QZOXUW7CJ5FDJV54TFUIWMOKLK7HDP2BTM6TYJA, seg_pos={0 0}, upload=440630]: Uploading piece... Dec 26 13:33:41.408958787 (502h4m41.280041832s) [noise=true , address=145.79.227.2:4122 , piece_id=YX4ZCATYJNBZAKJPPMMKK5NLHYWKDX27ZPYSJZ575A4D5TYONLNA, seg_pos={0 0}, upload=440630]: Uploading piece... Dec 26 13:33:42.886221623 (502h4m42.757304668s) [upload=440630]: Next returned nil segment Dec 26 13:33:42.886237353 (502h4m42.757320398s) [upload=440630]: Waiting for error group managing segments... Dec 26 13:33:43.290181168 (502h4m43.161264213s) [noise=true , address=145.79.227.17:4002 , piece_id=ENWLCA27UJZ27QZOXUW7CJ5FDJV54TFUIWMOKLK7HDP2BTM6TYJA, seg_pos={0 0}, upload=440630]: Done uploading piece. err: <nil> Dec 26 13:33:43.299594993 (502h4m43.170678038s) [noise=true , address=145.79.227.27:4708 , piece_id=M6765TMVYXJXOD2H6EGT7QQYDVQ6L54VUQVBGISKQHGWLPV5F5VQ, seg_pos={0 0}, upload=440630]: Done uploading piece. err: <nil> Dec 26 13:33:43.303927914 (502h4m43.175010959s) [noise=true , address=145.79.227.2:4122 , piece_id=YX4ZCATYJNBZAKJPPMMKK5NLHYWKDX27ZPYSJZ575A4D5TYONLNA, seg_pos={0 0}, upload=440630]: Done uploading piece. err: <nil> Dec 26 13:33:43.470369601 (502h4m43.341452636s) [noise=true , address=77.90.31.14:4679 , piece_id=NODI4AIAELCQUE2EM24YOMZPOO7HVEHNQZ7QPJQXX64HZRYMLP4A, seg_pos={0 0}, upload=440630]: Done uploading piece. err: <nil> Dec 26 13:33:43.564479173 (502h4m43.435562218s) [noise=true , address=45.91.116.11:4130 , piece_id=67MQFUMW6SPTIZAGSBTRYSZFSAY5OA7YWQC4SGYV6KTIRUGAP6YA, seg_pos={0 0}, upload=440630]: Done uploading piece. err: <nil> Dec 26 13:33:43.634048146 (502h4m43.505131191s) [noise=true , address=45.91.116.5:4313 , piece_id=Y5D2ZQN2RPA3UEVWYXSXNIESYSTTGNNFU3RHT7LM7PG4FDBU6OMQ, seg_pos={0 0}, upload=440630]: Done uploading piece. err: <nil> Dec 26 13:33:43.645510602 (502h4m43.516593647s) [noise=true , address=77.90.31.18:4816 , piece_id=UCXOYRVZHJXD3WTO4HDK6EI4XDNC5BJEIUTMQ7UMXD3XIDSMAVWQ, seg_pos={0 0}, upload=440630]: Done uploading piece. err: <nil> Dec 26 13:33:43.660701748 (502h4m43.531784783s) [noise=true , address=45.91.116.6:4995 , piece_id=TA2C5D6WCNHAZC66JJKDZAFECQ7L5RNPPMIXXG7VW5KHJ2UWSDQA, seg_pos={0 0}, upload=440630]: Done uploading piece. err: <nil> Dec 26 13:33:44.136510990 (502h4m44.007594035s) [noise=true , address=77.90.31.13:4638 , piece_id=TYLH3WHGROKFVJDKBD7YE4TV7SYDZNOWTVAQUA32HFABEGNCY57Q, seg_pos={0 0}, upload=440630]: Done uploading piece. err: <nil> Dec 26 13:33:44.212561443 (502h4m44.083644478s) [noise=true , address=145.79.227.30:4786 , piece_id=DJHTV4YTHNKTW3LY6D3NKVMDBH7E4I5IGITYDUTS25XCGRIPP4OA, seg_pos={0 0}, upload=440630]: Done uploading piece. err: <nil> Dec 26 13:33:44.215809659 (502h4m44.086892694s) [noise=true , address=77.90.31.17:4769 , piece_id=YTJJ5X7Y537C5EHWDMVXRFPDO5ALTR5JIE2M7EVT3HLAB4H7CQGA, seg_pos={0 0}, upload=440630]: Done uploading piece. err: <nil> Dec 26 13:33:44.242389070 (502h4m44.113472115s) [noise=true , address=145.79.227.44:4210 , piece_id=YYDUMRRH7SFDBGZVMCBCLHF65QMWF7TFGKZLSKN3ZNDPXOJQSNJA, seg_pos={0 0}, upload=440630]: Done uploading piece. err: <nil> Dec 26 13:33:44.344702805 (502h4m44.21578584s) [noise=true , address=145.79.227.40:4053 , piece_id=R6HUVGBWIZX7XB5L5UWZEDNOI6EQXOS2HAOG4UXMAN7UEYAH6GIQ, seg_pos={0 0}, upload=440630]: Done uploading piece. err: <nil> Dec 26 13:33:44.603185995 (502h4m44.47426903s) [noise=true , address=77.90.31.15:4729 , piece_id=EG7N7UWUQ6VIF7HHWKHKPNHBRP5JLDT6MYTF27TBVULB4UFDDDLQ, seg_pos={0 0}, upload=440630]: Done uploading piece. err: <nil> Dec 26 13:33:44.603204641 (502h4m44.474287676s) [seg_pos={0 0}, upload=440630]: Segment reached optimal threshold of 14 pieces. Dec 26 13:33:44.603386137 (502h4m44.474469172s) [noise=true , address=77.90.31.19:4848 , piece_id=52G73GXC4OODJLNJHWTWLIREZI5O364Y7ZMXSA2WXA57ALM3M3XA, seg_pos={0 0}, upload=440630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD): protocol: write tcp 188.166.225.170:46878->77.90.31.19:4848: use of closed network connection; write tcp 188.166.225.170:46878->77.90.31.19:4848: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46878->77.90.31.19:4848: use of closed network connection Dec 26 13:33:44.603402048 (502h4m44.474485083s) [noise=true , address=77.90.31.19:4848 , piece_id=52G73GXC4OODJLNJHWTWLIREZI5O364Y7ZMXSA2WXA57ALM3M3XA, seg_pos={0 0}, upload=440630]: Long tail cancellation due to optimal threshold Dec 26 13:33:44.603730072 (502h4m44.474813117s) [noise=true , address=64.49.13.169:4710 , piece_id=AKQFG2CMMF5TW3HJXWHL4QCGKKD55VW4VZONCCD5BVNXKVLNT6HA, seg_pos={0 0}, upload=440630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:45650->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:45650->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45650->64.49.13.169:4710: use of closed network connection Dec 26 13:33:44.603740231 (502h4m44.474823266s) [noise=true , address=64.49.13.169:4710 , piece_id=AKQFG2CMMF5TW3HJXWHL4QCGKKD55VW4VZONCCD5BVNXKVLNT6HA, seg_pos={0 0}, upload=440630]: Long tail cancellation due to optimal threshold Dec 26 13:33:44.603908942 (502h4m44.474991987s) [noise=true , address=145.79.227.11:4749 , piece_id=DKJJB7KRMZERWFK2NALFNNEXFQ2WI4RM674DFCCBOMTDFIFUL3BQ, seg_pos={0 0}, upload=440630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sRotj46WjNNsVciJDRvSEbrj8XHwshKRmMneedu6UH4egmZDm): protocol: expected piece hash; context canceled; EOF Dec 26 13:33:44.603915164 (502h4m44.474998209s) [noise=true , address=145.79.227.11:4749 , piece_id=DKJJB7KRMZERWFK2NALFNNEXFQ2WI4RM674DFCCBOMTDFIFUL3BQ, seg_pos={0 0}, upload=440630]: Long tail cancellation due to optimal threshold Dec 26 13:33:44.604069178 (502h4m44.475152213s) [noise=true , address=77.90.31.11:4572 , piece_id=EGDASBRSA5ZKKWRYLFCBRB5SZQRKJ65CYTLKWAJE7IOZAVUOPICQ, seg_pos={0 0}, upload=440630]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7): protocol: expected piece hash; context canceled; EOF Dec 26 13:33:44.604075259 (502h4m44.475158294s) [noise=true , address=77.90.31.11:4572 , piece_id=EGDASBRSA5ZKKWRYLFCBRB5SZQRKJ65CYTLKWAJE7IOZAVUOPICQ, seg_pos={0 0}, upload=440630]: Long tail cancellation due to optimal threshold Dec 26 13:33:44.604107100 (502h4m44.475190135s) [seg_pos={0 0}, upload=440630]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 13:33:44.604179508 (502h4m44.475262543s) [upload=440630]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:33:44.604184307 (502h4m44.475267342s) [upload=440630]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:33:44.906902750 (502h4m44.777985785s) [upload=440630]: Done uploading segments... Dec 26 13:33:44.906932617 (502h4m44.778015662s) [upload=440630]: Upload finished. err: <nil> Duration: 1m0.002750175s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [XX.X..X..X....XXXXXX...XX.....X...XX.X.XXX.XXXXX..X.X.X.X...] duration: 4.815735593s trace: https://jaeger.dev.storj.io/trace/f3cff59423f046e FAIL (2025-12-26 13:34:28 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.81575485s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [XXX.X..X..X....XXXXXX...XX.....X...XX.X.XXX.XXXXX..X.X.X.X..] duration: 5.001495066s trace: https://jaeger.dev.storj.io/trace/f3cff59423f046e FAIL (2025-12-26 13:33:49 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001520413s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 17 out of 60 is failed (threshold=3) [..X.XXX...X..X...................XX.X........X.XXX.X.X..X.X.] duration: 56.053711396s trace: https://jaeger.dev.storj.io/trace/7b24e960f69de86c FAIL (2025-12-26 13:33:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6 13:31:44.816127682 (502h2m44.687210727s) [noise=true , address=45.91.116.6:4993 , piece_id=QFTZZOP363PX7BSR3MFDIU3L3MZSFS5KNLGHMGBRIGWEGCNI62IA, seg_pos={0 0}, upload=440604]: Uploading piece... Dec 26 13:31:44.816505631 (502h2m44.687588676s) [noise=true , address=64.49.13.26:4952 , piece_id=TNVIOGDWDSOYCOGCGZ7QRHRHXXXZ4LOEWXM5YIDCSNRWTNSAQ4SQ, seg_pos={0 0}, upload=440604]: Uploading piece... Dec 26 13:31:44.816895414 (502h2m44.687978459s) [noise=true , address=77.90.31.14:4671 , piece_id=WRYJKE6EKGM6EMY6DF5YRVLTDJC3K5TTA3KQQX3X2EX4AWZUDY3A, seg_pos={0 0}, upload=440604]: Uploading piece... Dec 26 13:31:44.817273274 (502h2m44.688356319s) [noise=true , address=77.90.31.18:4821 , piece_id=YTQJ5PSTQCWAY6WXGR6URGAQXBQLP2BJCLYR3VIHOS33IMQBY6XQ, seg_pos={0 0}, upload=440604]: Uploading piece... Dec 26 13:31:44.817649750 (502h2m44.688732785s) [noise=true , address=64.49.13.169:4709 , piece_id=BKLKWI263ZV72RFAX6A4FLGUIAWWECVIKYDTNQBUNM4CW7U4W6FQ, seg_pos={0 0}, upload=440604]: Uploading piece... Dec 26 13:31:44.818196591 (502h2m44.689279636s) [noise=true , address=145.79.227.30:4766 , piece_id=U5C556LF2MD3ZWTLQH656APQXGW7QLJU44CE5QNTT76UXUNAWT5A, seg_pos={0 0}, upload=440604]: Uploading piece... Dec 26 13:31:44.818588247 (502h2m44.689671292s) [noise=true , address=77.90.31.17:4791 , piece_id=ZGNZ5M66HDQ3WANMBGF5WQAE7ZYFMO26GFE3ATWPCVXG2YGGVSSA, seg_pos={0 0}, upload=440604]: Uploading piece... Dec 26 13:31:44.818988880 (502h2m44.690071925s) [noise=true , address=77.90.31.12:4624 , piece_id=2GT5TLCMXSDAERGSDBMKILYNVO5GEIOTUPM3UW2W2QCYBIDXQX2Q, seg_pos={0 0}, upload=440604]: Uploading piece... Dec 26 13:31:44.819406075 (502h2m44.69048912s) [noise=true , address=145.79.227.21:4486 , piece_id=XNNMKB7N5MAJJH3CU7GMGTIXQA5Y5FK6OWYYIJYQSGBTKF36I27Q, seg_pos={0 0}, upload=440604]: Uploading piece... Dec 26 13:31:44.819911847 (502h2m44.690994882s) [noise=true , address=77.90.31.21:4901 , piece_id=3KEVG6TP4R6Q3OVM5YCEG44UHIXDX23HN33XT3HWJGRAJAT6CJQQ, seg_pos={0 0}, upload=440604]: Uploading piece... Dec 26 13:31:44.821346700 (502h2m44.692429745s) [noise=true , address=77.90.31.19:4873 , piece_id=XCZHDC7U65IFZ57CAIC5SIAEOQBMLD2ZRYL3WQMLQ5X4TF3NLVDQ, seg_pos={0 0}, upload=440604]: Uploading piece... Dec 26 13:31:44.823173639 (502h2m44.694256674s) [noise=true , address=145.79.227.40:4068 , piece_id=UIH5IBACI2RV3Q5LMJU2C5E57LPCSFHPMTE7JM5NGZTF6PBUUMQA, seg_pos={0 0}, upload=440604]: Uploading piece... Dec 26 13:31:44.824989045 (502h2m44.69607209s) [noise=true , address=77.90.31.11:4581 , piece_id=CQZAR3YWHXBERYXHIBONKBNLDGEVIVT2M3J3U7LMPGEXFXR7ZWCA, seg_pos={0 0}, upload=440604]: Uploading piece... Dec 26 13:31:44.826837775 (502h2m44.697920821s) [noise=true , address=145.79.227.46:4634 , piece_id=OBFVJUPK7FKEAOTCRD6Q63ELTQUKSE4NFDOFNPID5GVJQCWMULAA, seg_pos={0 0}, upload=440604]: Uploading piece... Dec 26 13:31:44.828469402 (502h2m44.699552448s) [noise=true , address=77.90.31.15:4718 , piece_id=LFQIAZGD222OSYLGDVPOIINJ56IYKCKREBP5VQSPQ4NA43CHMMIA, seg_pos={0 0}, upload=440604]: Uploading piece... Dec 26 13:31:44.830080821 (502h2m44.701163866s) [noise=true , address=45.91.116.12:4246 , piece_id=AN4KZZDCZ5ZYKLTTOMOGHOPYPFQ4JKXQJSAW2DBDPIDGLU7CBWJA, seg_pos={0 0}, upload=440604]: Uploading piece... Dec 26 13:31:46.286270189 (502h2m46.157353234s) [upload=440604]: Next returned nil segment Dec 26 13:31:46.286287351 (502h2m46.157370386s) [upload=440604]: Waiting for error group managing segments... Dec 26 13:31:46.682378534 (502h2m46.553461569s) [noise=true , address=145.79.227.21:4486 , piece_id=XNNMKB7N5MAJJH3CU7GMGTIXQA5Y5FK6OWYYIJYQSGBTKF36I27Q, seg_pos={0 0}, upload=440604]: Done uploading piece. err: <nil> Dec 26 13:31:46.717265995 (502h2m46.58834904s) [noise=true , address=64.49.13.26:4952 , piece_id=TNVIOGDWDSOYCOGCGZ7QRHRHXXXZ4LOEWXM5YIDCSNRWTNSAQ4SQ, seg_pos={0 0}, upload=440604]: Done uploading piece. err: <nil> Dec 26 13:31:46.724004849 (502h2m46.595087884s) [noise=true , address=145.79.227.46:4634 , piece_id=OBFVJUPK7FKEAOTCRD6Q63ELTQUKSE4NFDOFNPID5GVJQCWMULAA, seg_pos={0 0}, upload=440604]: Done uploading piece. err: <nil> Dec 26 13:31:46.909054056 (502h2m46.780137101s) [noise=true , address=45.91.116.6:4993 , piece_id=QFTZZOP363PX7BSR3MFDIU3L3MZSFS5KNLGHMGBRIGWEGCNI62IA, seg_pos={0 0}, upload=440604]: Done uploading piece. err: <nil> Dec 26 13:31:46.915928587 (502h2m46.787011622s) [noise=true , address=77.90.31.12:4624 , piece_id=2GT5TLCMXSDAERGSDBMKILYNVO5GEIOTUPM3UW2W2QCYBIDXQX2Q, seg_pos={0 0}, upload=440604]: Done uploading piece. err: <nil> Dec 26 13:31:46.925644368 (502h2m46.796727413s) [noise=true , address=45.91.116.12:4246 , piece_id=AN4KZZDCZ5ZYKLTTOMOGHOPYPFQ4JKXQJSAW2DBDPIDGLU7CBWJA, seg_pos={0 0}, upload=440604]: Done uploading piece. err: <nil> Dec 26 13:31:46.968121983 (502h2m46.839205018s) [noise=true , address=77.90.31.19:4873 , piece_id=XCZHDC7U65IFZ57CAIC5SIAEOQBMLD2ZRYL3WQMLQ5X4TF3NLVDQ, seg_pos={0 0}, upload=440604]: Done uploading piece. err: <nil> Dec 26 13:31:47.034042507 (502h2m46.905125542s) [noise=true , address=77.90.31.14:4671 , piece_id=WRYJKE6EKGM6EMY6DF5YRVLTDJC3K5TTA3KQQX3X2EX4AWZUDY3A, seg_pos={0 0}, upload=440604]: Done uploading piece. err: <nil> Dec 26 13:31:47.081485699 (502h2m46.952568734s) [noise=true , address=77.90.31.17:4791 , piece_id=ZGNZ5M66HDQ3WANMBGF5WQAE7ZYFMO26GFE3ATWPCVXG2YGGVSSA, seg_pos={0 0}, upload=440604]: Done uploading piece. err: <nil> Dec 26 13:31:47.096746518 (502h2m46.967829553s) [noise=true , address=77.90.31.18:4821 , piece_id=YTQJ5PSTQCWAY6WXGR6URGAQXBQLP2BJCLYR3VIHOS33IMQBY6XQ, seg_pos={0 0}, upload=440604]: Done uploading piece. err: <nil> Dec 26 13:31:47.419402115 (502h2m47.290485151s) [noise=true , address=77.90.31.11:4581 , piece_id=CQZAR3YWHXBERYXHIBONKBNLDGEVIVT2M3J3U7LMPGEXFXR7ZWCA, seg_pos={0 0}, upload=440604]: Done uploading piece. err: <nil> Dec 26 13:31:47.567966400 (502h2m47.439049435s) [noise=true , address=77.90.31.21:4901 , piece_id=3KEVG6TP4R6Q3OVM5YCEG44UHIXDX23HN33XT3HWJGRAJAT6CJQQ, seg_pos={0 0}, upload=440604]: Done uploading piece. err: <nil> Dec 26 13:31:47.612599410 (502h2m47.483682445s) [noise=true , address=145.79.227.40:4068 , piece_id=UIH5IBACI2RV3Q5LMJU2C5E57LPCSFHPMTE7JM5NGZTF6PBUUMQA, seg_pos={0 0}, upload=440604]: Done uploading piece. err: <nil> Dec 26 13:31:47.621689509 (502h2m47.492772554s) [noise=true , address=145.79.227.44:4208 , piece_id=4R6DJU5KAWELE2272DH2WANWGBHARDSLWTQX3IX7PRKWXSOVOMBA, seg_pos={0 0}, upload=440604]: Done uploading piece. err: <nil> Dec 26 13:31:47.621712262 (502h2m47.492795307s) [seg_pos={0 0}, upload=440604]: Segment reached optimal threshold of 14 pieces. Dec 26 13:31:47.621987286 (502h2m47.493070321s) [noise=true , address=77.90.31.15:4718 , piece_id=LFQIAZGD222OSYLGDVPOIINJ56IYKCKREBP5VQSPQ4NA43CHMMIA, seg_pos={0 0}, upload=440604]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC): protocol: expected piece hash; context canceled; EOF Dec 26 13:31:47.621995551 (502h2m47.493078587s) [noise=true , address=77.90.31.15:4718 , piece_id=LFQIAZGD222OSYLGDVPOIINJ56IYKCKREBP5VQSPQ4NA43CHMMIA, seg_pos={0 0}, upload=440604]: Long tail cancellation due to optimal threshold Dec 26 13:31:47.622059894 (502h2m47.493142929s) [noise=true , address=145.79.227.30:4766 , piece_id=U5C556LF2MD3ZWTLQH656APQXGW7QLJU44CE5QNTT76UXUNAWT5A, seg_pos={0 0}, upload=440604]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7): protocol: expected piece hash; context canceled; EOF Dec 26 13:31:47.622091474 (502h2m47.493174509s) [noise=true , address=145.79.227.30:4766 , piece_id=U5C556LF2MD3ZWTLQH656APQXGW7QLJU44CE5QNTT76UXUNAWT5A, seg_pos={0 0}, upload=440604]: Long tail cancellation due to optimal threshold Dec 26 13:31:47.622167118 (502h2m47.493250163s) [noise=true , address=45.91.116.13:4165 , piece_id=YVQVXYDQNS7PTWKJJXONL6HKUEEHE3XNWDBRN4HY7AXIQEHZRF3Q, seg_pos={0 0}, upload=440604]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg): protocol: expected piece hash; context canceled; EOF Dec 26 13:31:47.622177287 (502h2m47.493260332s) [noise=true , address=45.91.116.13:4165 , piece_id=YVQVXYDQNS7PTWKJJXONL6HKUEEHE3XNWDBRN4HY7AXIQEHZRF3Q, seg_pos={0 0}, upload=440604]: Long tail cancellation due to optimal threshold Dec 26 13:31:47.622388970 (502h2m47.493472015s) [noise=true , address=64.49.13.169:4709 , piece_id=BKLKWI263ZV72RFAX6A4FLGUIAWWECVIKYDTNQBUNM4CW7U4W6FQ, seg_pos={0 0}, upload=440604]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:59068->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:59068->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59068->64.49.13.169:4709: use of closed network connection Dec 26 13:31:47.622398117 (502h2m47.493481163s) [noise=true , address=64.49.13.169:4709 , piece_id=BKLKWI263ZV72RFAX6A4FLGUIAWWECVIKYDTNQBUNM4CW7U4W6FQ, seg_pos={0 0}, upload=440604]: Long tail cancellation due to optimal threshold Dec 26 13:31:47.622484422 (502h2m47.493567467s) [seg_pos={0 0}, upload=440604]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 13:31:47.622625010 (502h2m47.493708045s) [upload=440604]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:31:47.622630771 (502h2m47.493713806s) [upload=440604]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:31:47.921165387 (502h2m47.792248432s) [upload=440604]: Done uploading segments... Dec 26 13:31:47.921184063 (502h2m47.792267098s) [upload=440604]: Upload finished. err: <nil> Duration: 56.053730111s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [XXXX.X..X..X....XXXXXX...XX.....X...XX.X.XXX.XXXXX..X.X.X.X.] duration: 3.716212424s trace: https://jaeger.dev.storj.io/trace/f3cff59423f046e FAIL (2025-12-26 13:33:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.716248883s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.XX.....X..XXXXXX..X....X..X.X....X......XX.X.X......X.X.XX.] duration: 6.165518966s trace: https://jaeger.dev.storj.io/trace/3adcc84ba47e451a FAIL (2025-12-26 13:33:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: CGRXTVDO6DG5AJZKBSUGSU5FC6AYMB3CCPJ5KBWA3WLMY2RTWOLA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: RUZQITVVAEWPLKVLH5CI5WNADGBFORSKYHK55Y7FUZ27UGN5VGRA) context deadline exceeded ======LOGS====== Dec 26 13:31:47.254195340 (502h2m47.125278375s) [upload=440605]: Starting upload Dec 26 13:31:47.254252679 (502h2m47.125335714s) [upload=440605]: Done starting upload Dec 26 13:31:47.254349684 (502h2m47.125432719s) [upload=440605]: Uploading segments... Dec 26 13:31:47.254528093 (502h2m47.125611128s) [upload=440605]: Got next segment. Inline: false Dec 26 13:31:47.254586515 (502h2m47.12566955s) [upload=440605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 13:31:47.254591554 (502h2m47.125674589s) [upload=440605]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 13:31:47.501392270 (502h2m47.372475315s) [seg_pos={0 0}, upload=440605]: Begin upload segment... Dec 26 13:31:47.501443608 (502h2m47.372526643s) [seg_pos={0 0}, upload=440605]: Done begin upload segment. Dec 26 13:31:47.501543878 (502h2m47.372626913s) [noise=true , address=145.79.227.29:4942 , piece_id=K3SUP2L4FHQXANLILTGI7HGPQGMPZDTLGYMAX7TNNFCYMKZKYUUQ, seg_pos={0 0}, upload=440605]: Uploading piece... Dec 26 13:31:47.501577011 (502h2m47.372660056s) [noise=true , address=145.79.227.50:4373 , piece_id=WUNKFKTRNL57V3TM7A6GINB5ARCOFXHVZUX2KAVHBWOEUYBUV4NQ, seg_pos={0 0}, upload=440605]: Uploading piece... Dec 26 13:31:47.502497955 (502h2m47.373581s) [noise=true , address=45.91.116.14:4190 , piece_id=QIK4PXJY75TWMZZ5UEQYCLUNWC5ZEOVPDEVZJGIXVNV5F35ITS2A, seg_pos={0 0}, upload=440605]: Uploading piece... Dec 26 13:31:47.503634859 (502h2m47.374717894s) [noise=true , address=45.91.116.9:4262 , piece_id=CGRXTVDO6DG5AJZKBSUGSU5FC6AYMB3CCPJ5KBWA3WLMY2RTWOLA, seg_pos={0 0}, upload=440605]: Uploading piece... Dec 26 13:31:47.504688857 (502h2m47.375771892s) [noise=true , address=45.91.116.4:4939 , piece_id=NELRWH2C4J3O4OIWOSS5VU2LLIYWYTOQA5WL63G4DH2D46XA2WFQ, seg_pos={0 0}, upload=440605]: Uploading piece... Dec 26 13:31:47.508092849 (502h2m47.379175894s) [noise=true , address=145.79.227.4:4185 , piece_id=2JMFLYUOIQZTWZDAN2ILPIHLRCF6HLXDPGCQWGCQQY5NAJXO2LKQ, seg_pos={0 0}, upload=440605]: Uploading piece... Dec 26 13:31:47.508461961 (502h2m47.379544996s) [noise=true , address=45.91.116.13:4153 , piece_id=4FLRAJEEGF5SQM3GTAQXX3KGQK63KPIE4M3FPAVTV3LEZE322S2A, seg_pos={0 0}, upload=440605]: Uploading piece... Dec 26 13:31:47.509716891 (502h2m47.380799926s) [noise=true , address=145.79.227.11:4729 , piece_id=GR4LZT6QWLHJPCETXJCIO2PM4Y4R444PEN3XNWWNAMT2EZLJXWUQ, seg_pos={0 0}, upload=440605]: Uploading piece... Dec 26 13:31:47.510655138 (502h2m47.381738173s) [noise=true , address=77.90.31.11:4556 , piece_id=DAYZD7GMP6UQBD35HCIKEHOUB4JIOQSRME5YMQVL4OQHVLXUPHOA, seg_pos={0 0}, upload=440605]: Uploading piece... Dec 26 13:31:47.511527278 (502h2m47.382610323s) [noise=true , address=145.79.227.35:4481 , piece_id=4MBV44ZUSCJ2M6RIQCHNOM62YF4WD5EPKX7GQDCYZCTHF33CTKMQ, seg_pos={0 0}, upload=440605]: Uploading piece... Dec 26 13:31:47.511966445 (502h2m47.38304949s) [noise=true , address=145.79.227.3:4150 , piece_id=47YYKYZYQ25LHODTBVXT4BBHSOW65ZZIA5UEJKRGBZ3J4UWZ5YLQ, seg_pos={0 0}, upload=440605]: Uploading piece... Dec 26 13:31:47.512663252 (502h2m47.383746287s) [noise=true , address=145.79.227.15:4876 , piece_id=DNZJZ6FXFHI5AEAEFM2B2MTHMDZOD63S4IVWFUXO5W62XDJRPIXQ, seg_pos={0 0}, upload=440605]: Uploading piece... Dec 26 13:31:47.513394915 (502h2m47.38447795s) [noise=true , address=145.79.227.38:4978 , piece_id=KZ6PQUSJHPZIKXNWKFCBQ3HX46BZTDZFBWOJE5FYDJS7O4W7IXPA, seg_pos={0 0}, upload=440605]: Uploading piece... Dec 26 13:31:47.513676992 (502h2m47.384760027s) [noise=true , address=45.91.116.7:4011 , piece_id=RUZQITVVAEWPLKVLH5CI5WNADGBFORSKYHK55Y7FUZ27UGN5VGRA, seg_pos={0 0}, upload=440605]: Uploading piece... Dec 26 13:31:47.514945658 (502h2m47.386028693s) [noise=true , address=145.79.227.42:4121 , piece_id=6N7CBCXNXG4NHFV5BJDAU6MURT246JXONPRV66LBAKPA4PCQTBXQ, seg_pos={0 0}, upload=440605]: Uploading piece... Dec 26 13:31:47.515583893 (502h2m47.386666928s) [noise=true , address=145.79.227.13:4802 , piece_id=MICLSO7R5JHLK3HAOGZU3KDLAAJW7BUA7LCBMIZPOSGGWWKIA2MQ, seg_pos={0 0}, upload=440605]: Uploading piece... Dec 26 13:31:49.218629151 (502h2m49.089712186s) [upload=440605]: Next returned nil segment Dec 26 13:31:49.218645733 (502h2m49.089728779s) [upload=440605]: Waiting for error group managing segments... Dec 26 13:31:49.508684932 (502h2m49.379767977s) [noise=true , address=145.79.227.50:4373 , piece_id=WUNKFKTRNL57V3TM7A6GINB5ARCOFXHVZUX2KAVHBWOEUYBUV4NQ, seg_pos={0 0}, upload=440605]: Done uploading piece. err: <nil> Dec 26 13:31:49.508998109 (502h2m49.380081154s) [noise=true , address=145.79.227.29:4942 , piece_id=K3SUP2L4FHQXANLILTGI7HGPQGMPZDTLGYMAX7TNNFCYMKZKYUUQ, seg_pos={0 0}, upload=440605]: Done uploading piece. err: <nil> Dec 26 13:31:49.513481476 (502h2m49.384564521s) [noise=true , address=145.79.227.3:4150 , piece_id=47YYKYZYQ25LHODTBVXT4BBHSOW65ZZIA5UEJKRGBZ3J4UWZ5YLQ, seg_pos={0 0}, upload=440605]: Done uploading piece. err: <nil> Dec 26 13:31:49.514599204 (502h2m49.385682249s) [noise=true , address=145.79.227.4:4185 , piece_id=2JMFLYUOIQZTWZDAN2ILPIHLRCF6HLXDPGCQWGCQQY5NAJXO2LKQ, seg_pos={0 0}, upload=440605]: Done uploading piece. err: <nil> Dec 26 13:31:49.517470622 (502h2m49.388553657s) [noise=true , address=145.79.227.11:4729 , piece_id=GR4LZT6QWLHJPCETXJCIO2PM4Y4R444PEN3XNWWNAMT2EZLJXWUQ, seg_pos={0 0}, upload=440605]: Done uploading piece. err: <nil> Dec 26 13:31:49.519451264 (502h2m49.390534309s) [noise=true , address=145.79.227.35:4481 , piece_id=4MBV44ZUSCJ2M6RIQCHNOM62YF4WD5EPKX7GQDCYZCTHF33CTKMQ, seg_pos={0 0}, upload=440605]: Done uploading piece. err: <nil> Dec 26 13:31:49.536813664 (502h2m49.407896699s) [noise=true , address=145.79.227.42:4121 , piece_id=6N7CBCXNXG4NHFV5BJDAU6MURT246JXONPRV66LBAKPA4PCQTBXQ, seg_pos={0 0}, upload=440605]: Done uploading piece. err: <nil> Dec 26 13:31:49.539727703 (502h2m49.410810738s) [noise=true , address=145.79.227.38:4978 , piece_id=KZ6PQUSJHPZIKXNWKFCBQ3HX46BZTDZFBWOJE5FYDJS7O4W7IXPA, seg_pos={0 0}, upload=440605]: Done uploading piece. err: <nil> Dec 26 13:31:49.545676972 (502h2m49.416760017s) [noise=true , address=145.79.227.15:4876 , piece_id=DNZJZ6FXFHI5AEAEFM2B2MTHMDZOD63S4IVWFUXO5W62XDJRPIXQ, seg_pos={0 0}, upload=440605]: Done uploading piece. err: <nil> Dec 26 13:31:49.546573549 (502h2m49.417656594s) [noise=true , address=145.79.227.13:4802 , piece_id=MICLSO7R5JHLK3HAOGZU3KDLAAJW7BUA7LCBMIZPOSGGWWKIA2MQ, seg_pos={0 0}, upload=440605]: Done uploading piece. err: <nil> Dec 26 13:31:49.864392096 (502h2m49.735475131s) [noise=true , address=45.91.116.9:4262 , piece_id=CGRXTVDO6DG5AJZKBSUGSU5FC6AYMB3CCPJ5KBWA3WLMY2RTWOLA, seg_pos={0 0}, upload=440605]: Done uploading piece. err: <nil> Dec 26 13:31:49.873177955 (502h2m49.74426099s) [noise=true , address=45.91.116.14:4190 , piece_id=QIK4PXJY75TWMZZ5UEQYCLUNWC5ZEOVPDEVZJGIXVNV5F35ITS2A, seg_pos={0 0}, upload=440605]: Done uploading piece. err: <nil> Dec 26 13:31:50.093724567 (502h2m49.964807602s) [noise=true , address=45.91.116.7:4011 , piece_id=RUZQITVVAEWPLKVLH5CI5WNADGBFORSKYHK55Y7FUZ27UGN5VGRA, seg_pos={0 0}, upload=440605]: Done uploading piece. err: <nil> Dec 26 13:31:50.093748874 (502h2m49.964831919s) [seg_pos={0 0}, upload=440605]: Segment reached optimal threshold of 13 pieces. Dec 26 13:31:50.094073261 (502h2m49.965156306s) [noise=true , address=45.91.116.13:4153 , piece_id=4FLRAJEEGF5SQM3GTAQXX3KGQK63KPIE4M3FPAVTV3LEZE322S2A, seg_pos={0 0}, upload=440605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 26 13:31:50.094081727 (502h2m49.965164762s) [noise=true , address=45.91.116.13:4153 , piece_id=4FLRAJEEGF5SQM3GTAQXX3KGQK63KPIE4M3FPAVTV3LEZE322S2A, seg_pos={0 0}, upload=440605]: Long tail cancellation due to optimal threshold Dec 26 13:31:50.094344889 (502h2m49.965427924s) [noise=true , address=45.91.116.4:4939 , piece_id=NELRWH2C4J3O4OIWOSS5VU2LLIYWYTOQA5WL63G4DH2D46XA2WFQ, seg_pos={0 0}, upload=440605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 26 13:31:50.094352433 (502h2m49.965435478s) [noise=true , address=45.91.116.4:4939 , piece_id=NELRWH2C4J3O4OIWOSS5VU2LLIYWYTOQA5WL63G4DH2D46XA2WFQ, seg_pos={0 0}, upload=440605]: Long tail cancellation due to optimal threshold Dec 26 13:31:50.094486298 (502h2m49.965569343s) [noise=true , address=77.90.31.11:4556 , piece_id=DAYZD7GMP6UQBD35HCIKEHOUB4JIOQSRME5YMQVL4OQHVLXUPHOA, seg_pos={0 0}, upload=440605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 26 13:31:50.094493802 (502h2m49.965576847s) [noise=true , address=77.90.31.11:4556 , piece_id=DAYZD7GMP6UQBD35HCIKEHOUB4JIOQSRME5YMQVL4OQHVLXUPHOA, seg_pos={0 0}, upload=440605]: Long tail cancellation due to optimal threshold Dec 26 13:31:50.094516516 (502h2m49.965599561s) [seg_pos={0 0}, upload=440605]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 13:31:50.094592560 (502h2m49.965675595s) [upload=440605]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:31:50.094598702 (502h2m49.965681747s) [upload=440605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:31:50.390831514 (502h2m50.261914559s) [upload=440605]: Done uploading segments... Dec 26 13:31:50.390858165 (502h2m50.26194121s) [upload=440605]: Upload finished. err: <nil> Duration: 6.165557138s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [.XXXX.X..X..X....XXXXXX...XX.....X...XX.X.XXX.XXXXX..X.X.X.X] duration: 5.001803953s trace: https://jaeger.dev.storj.io/trace/f3cff59423f046e FAIL (2025-12-26 13:33:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001817698s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 17 out of 60 is failed (threshold=3) [...X.XXX...X..X...................XX.X........X.XXX.X.X..X.X] duration: 1m0.001710104s trace: https://jaeger.dev.storj.io/trace/7b24e960f69de86c FAIL (2025-12-26 13:32:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6 13:31:44.816127682 (502h2m44.687210727s) [noise=true , address=45.91.116.6:4993 , piece_id=QFTZZOP363PX7BSR3MFDIU3L3MZSFS5KNLGHMGBRIGWEGCNI62IA, seg_pos={0 0}, upload=440604]: Uploading piece... Dec 26 13:31:44.816505631 (502h2m44.687588676s) [noise=true , address=64.49.13.26:4952 , piece_id=TNVIOGDWDSOYCOGCGZ7QRHRHXXXZ4LOEWXM5YIDCSNRWTNSAQ4SQ, seg_pos={0 0}, upload=440604]: Uploading piece... Dec 26 13:31:44.816895414 (502h2m44.687978459s) [noise=true , address=77.90.31.14:4671 , piece_id=WRYJKE6EKGM6EMY6DF5YRVLTDJC3K5TTA3KQQX3X2EX4AWZUDY3A, seg_pos={0 0}, upload=440604]: Uploading piece... Dec 26 13:31:44.817273274 (502h2m44.688356319s) [noise=true , address=77.90.31.18:4821 , piece_id=YTQJ5PSTQCWAY6WXGR6URGAQXBQLP2BJCLYR3VIHOS33IMQBY6XQ, seg_pos={0 0}, upload=440604]: Uploading piece... Dec 26 13:31:44.817649750 (502h2m44.688732785s) [noise=true , address=64.49.13.169:4709 , piece_id=BKLKWI263ZV72RFAX6A4FLGUIAWWECVIKYDTNQBUNM4CW7U4W6FQ, seg_pos={0 0}, upload=440604]: Uploading piece... Dec 26 13:31:44.818196591 (502h2m44.689279636s) [noise=true , address=145.79.227.30:4766 , piece_id=U5C556LF2MD3ZWTLQH656APQXGW7QLJU44CE5QNTT76UXUNAWT5A, seg_pos={0 0}, upload=440604]: Uploading piece... Dec 26 13:31:44.818588247 (502h2m44.689671292s) [noise=true , address=77.90.31.17:4791 , piece_id=ZGNZ5M66HDQ3WANMBGF5WQAE7ZYFMO26GFE3ATWPCVXG2YGGVSSA, seg_pos={0 0}, upload=440604]: Uploading piece... Dec 26 13:31:44.818988880 (502h2m44.690071925s) [noise=true , address=77.90.31.12:4624 , piece_id=2GT5TLCMXSDAERGSDBMKILYNVO5GEIOTUPM3UW2W2QCYBIDXQX2Q, seg_pos={0 0}, upload=440604]: Uploading piece... Dec 26 13:31:44.819406075 (502h2m44.69048912s) [noise=true , address=145.79.227.21:4486 , piece_id=XNNMKB7N5MAJJH3CU7GMGTIXQA5Y5FK6OWYYIJYQSGBTKF36I27Q, seg_pos={0 0}, upload=440604]: Uploading piece... Dec 26 13:31:44.819911847 (502h2m44.690994882s) [noise=true , address=77.90.31.21:4901 , piece_id=3KEVG6TP4R6Q3OVM5YCEG44UHIXDX23HN33XT3HWJGRAJAT6CJQQ, seg_pos={0 0}, upload=440604]: Uploading piece... Dec 26 13:31:44.821346700 (502h2m44.692429745s) [noise=true , address=77.90.31.19:4873 , piece_id=XCZHDC7U65IFZ57CAIC5SIAEOQBMLD2ZRYL3WQMLQ5X4TF3NLVDQ, seg_pos={0 0}, upload=440604]: Uploading piece... Dec 26 13:31:44.823173639 (502h2m44.694256674s) [noise=true , address=145.79.227.40:4068 , piece_id=UIH5IBACI2RV3Q5LMJU2C5E57LPCSFHPMTE7JM5NGZTF6PBUUMQA, seg_pos={0 0}, upload=440604]: Uploading piece... Dec 26 13:31:44.824989045 (502h2m44.69607209s) [noise=true , address=77.90.31.11:4581 , piece_id=CQZAR3YWHXBERYXHIBONKBNLDGEVIVT2M3J3U7LMPGEXFXR7ZWCA, seg_pos={0 0}, upload=440604]: Uploading piece... Dec 26 13:31:44.826837775 (502h2m44.697920821s) [noise=true , address=145.79.227.46:4634 , piece_id=OBFVJUPK7FKEAOTCRD6Q63ELTQUKSE4NFDOFNPID5GVJQCWMULAA, seg_pos={0 0}, upload=440604]: Uploading piece... Dec 26 13:31:44.828469402 (502h2m44.699552448s) [noise=true , address=77.90.31.15:4718 , piece_id=LFQIAZGD222OSYLGDVPOIINJ56IYKCKREBP5VQSPQ4NA43CHMMIA, seg_pos={0 0}, upload=440604]: Uploading piece... Dec 26 13:31:44.830080821 (502h2m44.701163866s) [noise=true , address=45.91.116.12:4246 , piece_id=AN4KZZDCZ5ZYKLTTOMOGHOPYPFQ4JKXQJSAW2DBDPIDGLU7CBWJA, seg_pos={0 0}, upload=440604]: Uploading piece... Dec 26 13:31:46.286270189 (502h2m46.157353234s) [upload=440604]: Next returned nil segment Dec 26 13:31:46.286287351 (502h2m46.157370386s) [upload=440604]: Waiting for error group managing segments... Dec 26 13:31:46.682378534 (502h2m46.553461569s) [noise=true , address=145.79.227.21:4486 , piece_id=XNNMKB7N5MAJJH3CU7GMGTIXQA5Y5FK6OWYYIJYQSGBTKF36I27Q, seg_pos={0 0}, upload=440604]: Done uploading piece. err: <nil> Dec 26 13:31:46.717265995 (502h2m46.58834904s) [noise=true , address=64.49.13.26:4952 , piece_id=TNVIOGDWDSOYCOGCGZ7QRHRHXXXZ4LOEWXM5YIDCSNRWTNSAQ4SQ, seg_pos={0 0}, upload=440604]: Done uploading piece. err: <nil> Dec 26 13:31:46.724004849 (502h2m46.595087884s) [noise=true , address=145.79.227.46:4634 , piece_id=OBFVJUPK7FKEAOTCRD6Q63ELTQUKSE4NFDOFNPID5GVJQCWMULAA, seg_pos={0 0}, upload=440604]: Done uploading piece. err: <nil> Dec 26 13:31:46.909054056 (502h2m46.780137101s) [noise=true , address=45.91.116.6:4993 , piece_id=QFTZZOP363PX7BSR3MFDIU3L3MZSFS5KNLGHMGBRIGWEGCNI62IA, seg_pos={0 0}, upload=440604]: Done uploading piece. err: <nil> Dec 26 13:31:46.915928587 (502h2m46.787011622s) [noise=true , address=77.90.31.12:4624 , piece_id=2GT5TLCMXSDAERGSDBMKILYNVO5GEIOTUPM3UW2W2QCYBIDXQX2Q, seg_pos={0 0}, upload=440604]: Done uploading piece. err: <nil> Dec 26 13:31:46.925644368 (502h2m46.796727413s) [noise=true , address=45.91.116.12:4246 , piece_id=AN4KZZDCZ5ZYKLTTOMOGHOPYPFQ4JKXQJSAW2DBDPIDGLU7CBWJA, seg_pos={0 0}, upload=440604]: Done uploading piece. err: <nil> Dec 26 13:31:46.968121983 (502h2m46.839205018s) [noise=true , address=77.90.31.19:4873 , piece_id=XCZHDC7U65IFZ57CAIC5SIAEOQBMLD2ZRYL3WQMLQ5X4TF3NLVDQ, seg_pos={0 0}, upload=440604]: Done uploading piece. err: <nil> Dec 26 13:31:47.034042507 (502h2m46.905125542s) [noise=true , address=77.90.31.14:4671 , piece_id=WRYJKE6EKGM6EMY6DF5YRVLTDJC3K5TTA3KQQX3X2EX4AWZUDY3A, seg_pos={0 0}, upload=440604]: Done uploading piece. err: <nil> Dec 26 13:31:47.081485699 (502h2m46.952568734s) [noise=true , address=77.90.31.17:4791 , piece_id=ZGNZ5M66HDQ3WANMBGF5WQAE7ZYFMO26GFE3ATWPCVXG2YGGVSSA, seg_pos={0 0}, upload=440604]: Done uploading piece. err: <nil> Dec 26 13:31:47.096746518 (502h2m46.967829553s) [noise=true , address=77.90.31.18:4821 , piece_id=YTQJ5PSTQCWAY6WXGR6URGAQXBQLP2BJCLYR3VIHOS33IMQBY6XQ, seg_pos={0 0}, upload=440604]: Done uploading piece. err: <nil> Dec 26 13:31:47.419402115 (502h2m47.290485151s) [noise=true , address=77.90.31.11:4581 , piece_id=CQZAR3YWHXBERYXHIBONKBNLDGEVIVT2M3J3U7LMPGEXFXR7ZWCA, seg_pos={0 0}, upload=440604]: Done uploading piece. err: <nil> Dec 26 13:31:47.567966400 (502h2m47.439049435s) [noise=true , address=77.90.31.21:4901 , piece_id=3KEVG6TP4R6Q3OVM5YCEG44UHIXDX23HN33XT3HWJGRAJAT6CJQQ, seg_pos={0 0}, upload=440604]: Done uploading piece. err: <nil> Dec 26 13:31:47.612599410 (502h2m47.483682445s) [noise=true , address=145.79.227.40:4068 , piece_id=UIH5IBACI2RV3Q5LMJU2C5E57LPCSFHPMTE7JM5NGZTF6PBUUMQA, seg_pos={0 0}, upload=440604]: Done uploading piece. err: <nil> Dec 26 13:31:47.621689509 (502h2m47.492772554s) [noise=true , address=145.79.227.44:4208 , piece_id=4R6DJU5KAWELE2272DH2WANWGBHARDSLWTQX3IX7PRKWXSOVOMBA, seg_pos={0 0}, upload=440604]: Done uploading piece. err: <nil> Dec 26 13:31:47.621712262 (502h2m47.492795307s) [seg_pos={0 0}, upload=440604]: Segment reached optimal threshold of 14 pieces. Dec 26 13:31:47.621987286 (502h2m47.493070321s) [noise=true , address=77.90.31.15:4718 , piece_id=LFQIAZGD222OSYLGDVPOIINJ56IYKCKREBP5VQSPQ4NA43CHMMIA, seg_pos={0 0}, upload=440604]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC): protocol: expected piece hash; context canceled; EOF Dec 26 13:31:47.621995551 (502h2m47.493078587s) [noise=true , address=77.90.31.15:4718 , piece_id=LFQIAZGD222OSYLGDVPOIINJ56IYKCKREBP5VQSPQ4NA43CHMMIA, seg_pos={0 0}, upload=440604]: Long tail cancellation due to optimal threshold Dec 26 13:31:47.622059894 (502h2m47.493142929s) [noise=true , address=145.79.227.30:4766 , piece_id=U5C556LF2MD3ZWTLQH656APQXGW7QLJU44CE5QNTT76UXUNAWT5A, seg_pos={0 0}, upload=440604]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7): protocol: expected piece hash; context canceled; EOF Dec 26 13:31:47.622091474 (502h2m47.493174509s) [noise=true , address=145.79.227.30:4766 , piece_id=U5C556LF2MD3ZWTLQH656APQXGW7QLJU44CE5QNTT76UXUNAWT5A, seg_pos={0 0}, upload=440604]: Long tail cancellation due to optimal threshold Dec 26 13:31:47.622167118 (502h2m47.493250163s) [noise=true , address=45.91.116.13:4165 , piece_id=YVQVXYDQNS7PTWKJJXONL6HKUEEHE3XNWDBRN4HY7AXIQEHZRF3Q, seg_pos={0 0}, upload=440604]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg): protocol: expected piece hash; context canceled; EOF Dec 26 13:31:47.622177287 (502h2m47.493260332s) [noise=true , address=45.91.116.13:4165 , piece_id=YVQVXYDQNS7PTWKJJXONL6HKUEEHE3XNWDBRN4HY7AXIQEHZRF3Q, seg_pos={0 0}, upload=440604]: Long tail cancellation due to optimal threshold Dec 26 13:31:47.622388970 (502h2m47.493472015s) [noise=true , address=64.49.13.169:4709 , piece_id=BKLKWI263ZV72RFAX6A4FLGUIAWWECVIKYDTNQBUNM4CW7U4W6FQ, seg_pos={0 0}, upload=440604]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:59068->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:59068->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59068->64.49.13.169:4709: use of closed network connection Dec 26 13:31:47.622398117 (502h2m47.493481163s) [noise=true , address=64.49.13.169:4709 , piece_id=BKLKWI263ZV72RFAX6A4FLGUIAWWECVIKYDTNQBUNM4CW7U4W6FQ, seg_pos={0 0}, upload=440604]: Long tail cancellation due to optimal threshold Dec 26 13:31:47.622484422 (502h2m47.493567467s) [seg_pos={0 0}, upload=440604]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 13:31:47.622625010 (502h2m47.493708045s) [upload=440604]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:31:47.622630771 (502h2m47.493713806s) [upload=440604]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:31:47.921165387 (502h2m47.792248432s) [upload=440604]: Done uploading segments... Dec 26 13:31:47.921184063 (502h2m47.792267098s) [upload=440604]: Upload finished. err: <nil> Duration: 1m0.001728499s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..XX.....X..XXXXXX..X....X..X.X....X......XX.X.X......X.X.XX] duration: 30.002953691s trace: https://jaeger.dev.storj.io/trace/3adcc84ba47e451a FAIL (2025-12-26 13:32:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: CGRXTVDO6DG5AJZKBSUGSU5FC6AYMB3CCPJ5KBWA3WLMY2RTWOLA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: RUZQITVVAEWPLKVLH5CI5WNADGBFORSKYHK55Y7FUZ27UGN5VGRA) context deadline exceeded ======LOGS====== Dec 26 13:31:47.254195340 (502h2m47.125278375s) [upload=440605]: Starting upload Dec 26 13:31:47.254252679 (502h2m47.125335714s) [upload=440605]: Done starting upload Dec 26 13:31:47.254349684 (502h2m47.125432719s) [upload=440605]: Uploading segments... Dec 26 13:31:47.254528093 (502h2m47.125611128s) [upload=440605]: Got next segment. Inline: false Dec 26 13:31:47.254586515 (502h2m47.12566955s) [upload=440605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 13:31:47.254591554 (502h2m47.125674589s) [upload=440605]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 13:31:47.501392270 (502h2m47.372475315s) [seg_pos={0 0}, upload=440605]: Begin upload segment... Dec 26 13:31:47.501443608 (502h2m47.372526643s) [seg_pos={0 0}, upload=440605]: Done begin upload segment. Dec 26 13:31:47.501543878 (502h2m47.372626913s) [noise=true , address=145.79.227.29:4942 , piece_id=K3SUP2L4FHQXANLILTGI7HGPQGMPZDTLGYMAX7TNNFCYMKZKYUUQ, seg_pos={0 0}, upload=440605]: Uploading piece... Dec 26 13:31:47.501577011 (502h2m47.372660056s) [noise=true , address=145.79.227.50:4373 , piece_id=WUNKFKTRNL57V3TM7A6GINB5ARCOFXHVZUX2KAVHBWOEUYBUV4NQ, seg_pos={0 0}, upload=440605]: Uploading piece... Dec 26 13:31:47.502497955 (502h2m47.373581s) [noise=true , address=45.91.116.14:4190 , piece_id=QIK4PXJY75TWMZZ5UEQYCLUNWC5ZEOVPDEVZJGIXVNV5F35ITS2A, seg_pos={0 0}, upload=440605]: Uploading piece... Dec 26 13:31:47.503634859 (502h2m47.374717894s) [noise=true , address=45.91.116.9:4262 , piece_id=CGRXTVDO6DG5AJZKBSUGSU5FC6AYMB3CCPJ5KBWA3WLMY2RTWOLA, seg_pos={0 0}, upload=440605]: Uploading piece... Dec 26 13:31:47.504688857 (502h2m47.375771892s) [noise=true , address=45.91.116.4:4939 , piece_id=NELRWH2C4J3O4OIWOSS5VU2LLIYWYTOQA5WL63G4DH2D46XA2WFQ, seg_pos={0 0}, upload=440605]: Uploading piece... Dec 26 13:31:47.508092849 (502h2m47.379175894s) [noise=true , address=145.79.227.4:4185 , piece_id=2JMFLYUOIQZTWZDAN2ILPIHLRCF6HLXDPGCQWGCQQY5NAJXO2LKQ, seg_pos={0 0}, upload=440605]: Uploading piece... Dec 26 13:31:47.508461961 (502h2m47.379544996s) [noise=true , address=45.91.116.13:4153 , piece_id=4FLRAJEEGF5SQM3GTAQXX3KGQK63KPIE4M3FPAVTV3LEZE322S2A, seg_pos={0 0}, upload=440605]: Uploading piece... Dec 26 13:31:47.509716891 (502h2m47.380799926s) [noise=true , address=145.79.227.11:4729 , piece_id=GR4LZT6QWLHJPCETXJCIO2PM4Y4R444PEN3XNWWNAMT2EZLJXWUQ, seg_pos={0 0}, upload=440605]: Uploading piece... Dec 26 13:31:47.510655138 (502h2m47.381738173s) [noise=true , address=77.90.31.11:4556 , piece_id=DAYZD7GMP6UQBD35HCIKEHOUB4JIOQSRME5YMQVL4OQHVLXUPHOA, seg_pos={0 0}, upload=440605]: Uploading piece... Dec 26 13:31:47.511527278 (502h2m47.382610323s) [noise=true , address=145.79.227.35:4481 , piece_id=4MBV44ZUSCJ2M6RIQCHNOM62YF4WD5EPKX7GQDCYZCTHF33CTKMQ, seg_pos={0 0}, upload=440605]: Uploading piece... Dec 26 13:31:47.511966445 (502h2m47.38304949s) [noise=true , address=145.79.227.3:4150 , piece_id=47YYKYZYQ25LHODTBVXT4BBHSOW65ZZIA5UEJKRGBZ3J4UWZ5YLQ, seg_pos={0 0}, upload=440605]: Uploading piece... Dec 26 13:31:47.512663252 (502h2m47.383746287s) [noise=true , address=145.79.227.15:4876 , piece_id=DNZJZ6FXFHI5AEAEFM2B2MTHMDZOD63S4IVWFUXO5W62XDJRPIXQ, seg_pos={0 0}, upload=440605]: Uploading piece... Dec 26 13:31:47.513394915 (502h2m47.38447795s) [noise=true , address=145.79.227.38:4978 , piece_id=KZ6PQUSJHPZIKXNWKFCBQ3HX46BZTDZFBWOJE5FYDJS7O4W7IXPA, seg_pos={0 0}, upload=440605]: Uploading piece... Dec 26 13:31:47.513676992 (502h2m47.384760027s) [noise=true , address=45.91.116.7:4011 , piece_id=RUZQITVVAEWPLKVLH5CI5WNADGBFORSKYHK55Y7FUZ27UGN5VGRA, seg_pos={0 0}, upload=440605]: Uploading piece... Dec 26 13:31:47.514945658 (502h2m47.386028693s) [noise=true , address=145.79.227.42:4121 , piece_id=6N7CBCXNXG4NHFV5BJDAU6MURT246JXONPRV66LBAKPA4PCQTBXQ, seg_pos={0 0}, upload=440605]: Uploading piece... Dec 26 13:31:47.515583893 (502h2m47.386666928s) [noise=true , address=145.79.227.13:4802 , piece_id=MICLSO7R5JHLK3HAOGZU3KDLAAJW7BUA7LCBMIZPOSGGWWKIA2MQ, seg_pos={0 0}, upload=440605]: Uploading piece... Dec 26 13:31:49.218629151 (502h2m49.089712186s) [upload=440605]: Next returned nil segment Dec 26 13:31:49.218645733 (502h2m49.089728779s) [upload=440605]: Waiting for error group managing segments... Dec 26 13:31:49.508684932 (502h2m49.379767977s) [noise=true , address=145.79.227.50:4373 , piece_id=WUNKFKTRNL57V3TM7A6GINB5ARCOFXHVZUX2KAVHBWOEUYBUV4NQ, seg_pos={0 0}, upload=440605]: Done uploading piece. err: <nil> Dec 26 13:31:49.508998109 (502h2m49.380081154s) [noise=true , address=145.79.227.29:4942 , piece_id=K3SUP2L4FHQXANLILTGI7HGPQGMPZDTLGYMAX7TNNFCYMKZKYUUQ, seg_pos={0 0}, upload=440605]: Done uploading piece. err: <nil> Dec 26 13:31:49.513481476 (502h2m49.384564521s) [noise=true , address=145.79.227.3:4150 , piece_id=47YYKYZYQ25LHODTBVXT4BBHSOW65ZZIA5UEJKRGBZ3J4UWZ5YLQ, seg_pos={0 0}, upload=440605]: Done uploading piece. err: <nil> Dec 26 13:31:49.514599204 (502h2m49.385682249s) [noise=true , address=145.79.227.4:4185 , piece_id=2JMFLYUOIQZTWZDAN2ILPIHLRCF6HLXDPGCQWGCQQY5NAJXO2LKQ, seg_pos={0 0}, upload=440605]: Done uploading piece. err: <nil> Dec 26 13:31:49.517470622 (502h2m49.388553657s) [noise=true , address=145.79.227.11:4729 , piece_id=GR4LZT6QWLHJPCETXJCIO2PM4Y4R444PEN3XNWWNAMT2EZLJXWUQ, seg_pos={0 0}, upload=440605]: Done uploading piece. err: <nil> Dec 26 13:31:49.519451264 (502h2m49.390534309s) [noise=true , address=145.79.227.35:4481 , piece_id=4MBV44ZUSCJ2M6RIQCHNOM62YF4WD5EPKX7GQDCYZCTHF33CTKMQ, seg_pos={0 0}, upload=440605]: Done uploading piece. err: <nil> Dec 26 13:31:49.536813664 (502h2m49.407896699s) [noise=true , address=145.79.227.42:4121 , piece_id=6N7CBCXNXG4NHFV5BJDAU6MURT246JXONPRV66LBAKPA4PCQTBXQ, seg_pos={0 0}, upload=440605]: Done uploading piece. err: <nil> Dec 26 13:31:49.539727703 (502h2m49.410810738s) [noise=true , address=145.79.227.38:4978 , piece_id=KZ6PQUSJHPZIKXNWKFCBQ3HX46BZTDZFBWOJE5FYDJS7O4W7IXPA, seg_pos={0 0}, upload=440605]: Done uploading piece. err: <nil> Dec 26 13:31:49.545676972 (502h2m49.416760017s) [noise=true , address=145.79.227.15:4876 , piece_id=DNZJZ6FXFHI5AEAEFM2B2MTHMDZOD63S4IVWFUXO5W62XDJRPIXQ, seg_pos={0 0}, upload=440605]: Done uploading piece. err: <nil> Dec 26 13:31:49.546573549 (502h2m49.417656594s) [noise=true , address=145.79.227.13:4802 , piece_id=MICLSO7R5JHLK3HAOGZU3KDLAAJW7BUA7LCBMIZPOSGGWWKIA2MQ, seg_pos={0 0}, upload=440605]: Done uploading piece. err: <nil> Dec 26 13:31:49.864392096 (502h2m49.735475131s) [noise=true , address=45.91.116.9:4262 , piece_id=CGRXTVDO6DG5AJZKBSUGSU5FC6AYMB3CCPJ5KBWA3WLMY2RTWOLA, seg_pos={0 0}, upload=440605]: Done uploading piece. err: <nil> Dec 26 13:31:49.873177955 (502h2m49.74426099s) [noise=true , address=45.91.116.14:4190 , piece_id=QIK4PXJY75TWMZZ5UEQYCLUNWC5ZEOVPDEVZJGIXVNV5F35ITS2A, seg_pos={0 0}, upload=440605]: Done uploading piece. err: <nil> Dec 26 13:31:50.093724567 (502h2m49.964807602s) [noise=true , address=45.91.116.7:4011 , piece_id=RUZQITVVAEWPLKVLH5CI5WNADGBFORSKYHK55Y7FUZ27UGN5VGRA, seg_pos={0 0}, upload=440605]: Done uploading piece. err: <nil> Dec 26 13:31:50.093748874 (502h2m49.964831919s) [seg_pos={0 0}, upload=440605]: Segment reached optimal threshold of 13 pieces. Dec 26 13:31:50.094073261 (502h2m49.965156306s) [noise=true , address=45.91.116.13:4153 , piece_id=4FLRAJEEGF5SQM3GTAQXX3KGQK63KPIE4M3FPAVTV3LEZE322S2A, seg_pos={0 0}, upload=440605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 26 13:31:50.094081727 (502h2m49.965164762s) [noise=true , address=45.91.116.13:4153 , piece_id=4FLRAJEEGF5SQM3GTAQXX3KGQK63KPIE4M3FPAVTV3LEZE322S2A, seg_pos={0 0}, upload=440605]: Long tail cancellation due to optimal threshold Dec 26 13:31:50.094344889 (502h2m49.965427924s) [noise=true , address=45.91.116.4:4939 , piece_id=NELRWH2C4J3O4OIWOSS5VU2LLIYWYTOQA5WL63G4DH2D46XA2WFQ, seg_pos={0 0}, upload=440605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 26 13:31:50.094352433 (502h2m49.965435478s) [noise=true , address=45.91.116.4:4939 , piece_id=NELRWH2C4J3O4OIWOSS5VU2LLIYWYTOQA5WL63G4DH2D46XA2WFQ, seg_pos={0 0}, upload=440605]: Long tail cancellation due to optimal threshold Dec 26 13:31:50.094486298 (502h2m49.965569343s) [noise=true , address=77.90.31.11:4556 , piece_id=DAYZD7GMP6UQBD35HCIKEHOUB4JIOQSRME5YMQVL4OQHVLXUPHOA, seg_pos={0 0}, upload=440605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 26 13:31:50.094493802 (502h2m49.965576847s) [noise=true , address=77.90.31.11:4556 , piece_id=DAYZD7GMP6UQBD35HCIKEHOUB4JIOQSRME5YMQVL4OQHVLXUPHOA, seg_pos={0 0}, upload=440605]: Long tail cancellation due to optimal threshold Dec 26 13:31:50.094516516 (502h2m49.965599561s) [seg_pos={0 0}, upload=440605]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 13:31:50.094592560 (502h2m49.965675595s) [upload=440605]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:31:50.094598702 (502h2m49.965681747s) [upload=440605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:31:50.390831514 (502h2m50.261914559s) [upload=440605]: Done uploading segments... Dec 26 13:31:50.390858165 (502h2m50.26194121s) [upload=440605]: Upload finished. err: <nil> Duration: 30.003041848s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [X.XXXX.X..X..X....XXXXXX...XX.....X...XX.X.XXX.XXXXX..X.X.X.] duration: 5.001273764s trace: https://jaeger.dev.storj.io/trace/c1b9573f2752f26 FAIL (2025-12-26 13:32:02 UTC) orbiter warning: context deadline exceeded Duration: 5.001328168s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X..XX.....X..XXXXXX..X....X..X.X....X......XX.X.X......X.X.X] duration: 30.005365587s trace: https://jaeger.dev.storj.io/trace/537460ef749670a7 FAIL (2025-12-26 13:31:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 13:31:02.709507191 (502h2m2.580590236s) [upload=440596]: Starting upload Dec 26 13:31:02.709566463 (502h2m2.580649508s) [upload=440596]: Done starting upload Dec 26 13:31:02.709724455 (502h2m2.5808075s) [upload=440596]: Uploading segments... Dec 26 13:31:02.709915117 (502h2m2.580998162s) [upload=440596]: Got next segment. Inline: false Dec 26 13:31:02.709970894 (502h2m2.581053939s) [upload=440596]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 13:31:02.709978768 (502h2m2.581061814s) [upload=440596]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 13:31:02.958814810 (502h2m2.829897855s) [seg_pos={0 0}, upload=440596]: Begin upload segment... Dec 26 13:31:02.958863182 (502h2m2.829946227s) [seg_pos={0 0}, upload=440596]: Done begin upload segment. Dec 26 13:31:02.958897958 (502h2m2.829980993s) [noise=true , address=145.79.227.42:4121 , piece_id=QMDX2HM25FEYRGSZEJD3ER25QEG4ILL7M6MT4OMUQNH564P6NWYQ, seg_pos={0 0}, upload=440596]: Uploading piece... Dec 26 13:31:02.959772854 (502h2m2.830855899s) [noise=true , address=145.79.227.45:4230 , piece_id=7JFEWPGLA2URVDPDXZZM3GY4JFAXO3VENBZ3ACVS4C2XZQA2KIOQ, seg_pos={0 0}, upload=440596]: Uploading piece... Dec 26 13:31:02.960338551 (502h2m2.831421596s) [noise=true , address=45.91.116.13:4154 , piece_id=X3PGD4LFI2QXAPXII3P3YM2AQRVDFLN4FP2HK245OECTCQ6RAAUQ, seg_pos={0 0}, upload=440596]: Uploading piece... Dec 26 13:31:02.961042612 (502h2m2.832125657s) [noise=true , address=77.90.31.15:4700 , piece_id=BEVWL5ON5FOZQLQAZSEINWBMKDWDE7KRPN66GFSK2M5AXDACFMJA, seg_pos={0 0}, upload=440596]: Uploading piece... Dec 26 13:31:02.961501696 (502h2m2.832584741s) [noise=true , address=45.91.116.8:4047 , piece_id=ZCL6DLT2KZTY2U67GF3Q7NV5WRVM7AFWRMH4PEW3XGJTLA522TOQ, seg_pos={0 0}, upload=440596]: Uploading piece... Dec 26 13:31:02.962185047 (502h2m2.833268092s) [noise=true , address=145.79.227.16:4910 , piece_id=FQCIMCSMA76G5ATBMFKXOUEF2AV75UOT6H6FGPTQTP6YCK43KOPQ, seg_pos={0 0}, upload=440596]: Uploading piece... Dec 26 13:31:02.963668963 (502h2m2.834752008s) [noise=true , address=145.79.227.19:4078 , piece_id=KMAG4V52NQP2CBDCUTDMJVCUQW4TYBGX2O3FC6THXD3T5MIXPMHA, seg_pos={0 0}, upload=440596]: Uploading piece... Dec 26 13:31:02.965236207 (502h2m2.836319252s) [noise=true , address=145.79.227.51:4410 , piece_id=D25U4HOV26H2OKXARQJQFTZRGYRWC5UJH3E4IU3B73LTWL3SNAGA, seg_pos={0 0}, upload=440596]: Uploading piece... Dec 26 13:31:02.967254570 (502h2m2.838337605s) [noise=true , address=77.90.31.21:4878 , piece_id=C2P247WP3OZMADOYB4WFN4BPQICO2PQVDZ7FE64TZ6HH7ONE7WVQ, seg_pos={0 0}, upload=440596]: Uploading piece... Dec 26 13:31:02.968486275 (502h2m2.83956932s) [noise=true , address=145.79.227.26:4654 , piece_id=YZTKQ3L2Z22GDXF4SJ224AS4IEDFPPVHHD5QKCDOT7ZIFWDVSA6Q, seg_pos={0 0}, upload=440596]: Uploading piece... Dec 26 13:31:02.969507981 (502h2m2.840591027s) [noise=true , address=145.79.227.13:4804 , piece_id=VR4HBYAA2URAYZ4PE3GZBP4LKDIPPY5GBSVBY4ZHC6CDWCJEAOUQ, seg_pos={0 0}, upload=440596]: Uploading piece... Dec 26 13:31:02.970080521 (502h2m2.841163566s) [noise=true , address=45.91.116.11:4120 , piece_id=UIZ7STB3DVXXHEJ6VE5DLUKDHBCQ37SIQO5EQHHXWMXAOWOVL2JA, seg_pos={0 0}, upload=440596]: Uploading piece... Dec 26 13:31:02.970740528 (502h2m2.841823573s) [noise=true , address=145.79.227.49:4339 , piece_id=BKPQSJX5232UELVCLP2XRWYHDFUBQGQ6BGI6ST5VQULEFM4WMIDA, seg_pos={0 0}, upload=440596]: Uploading piece... Dec 26 13:31:02.971467853 (502h2m2.842550898s) [noise=true , address=145.79.227.46:4601 , piece_id=7LMVQBMEV4HAC55YX677QP6YYRRXRBJ6JMQEZMPVBOP76DKNASRA, seg_pos={0 0}, upload=440596]: Uploading piece... Dec 26 13:31:02.972939295 (502h2m2.84402233s) [noise=true , address=45.91.116.12:4225 , piece_id=GEHXFI6R66VTL4J22OUJEBURZHVA6T6GBYRWVBDSELZDP5DFNKNQ, seg_pos={0 0}, upload=440596]: Uploading piece... Dec 26 13:31:02.974774179 (502h2m2.845857214s) [noise=true , address=145.79.227.17:4970 , piece_id=SCM4BGWZDM32LWDRWUY5OFVMXO27DEXLGSGOR42RGNGSTQZ6ZHIQ, seg_pos={0 0}, upload=440596]: Uploading piece... Dec 26 13:31:04.674934804 (502h2m4.546017849s) [upload=440596]: Next returned nil segment Dec 26 13:31:04.674947017 (502h2m4.546030062s) [upload=440596]: Waiting for error group managing segments... Dec 26 13:31:04.960299723 (502h2m4.831382758s) [noise=true , address=145.79.227.51:4410 , piece_id=D25U4HOV26H2OKXARQJQFTZRGYRWC5UJH3E4IU3B73LTWL3SNAGA, seg_pos={0 0}, upload=440596]: Done uploading piece. err: <nil> Dec 26 13:31:04.960846995 (502h2m4.83193003s) [noise=true , address=145.79.227.19:4078 , piece_id=KMAG4V52NQP2CBDCUTDMJVCUQW4TYBGX2O3FC6THXD3T5MIXPMHA, seg_pos={0 0}, upload=440596]: Done uploading piece. err: <nil> Dec 26 13:31:04.969069422 (502h2m4.840152467s) [noise=true , address=145.79.227.49:4339 , piece_id=BKPQSJX5232UELVCLP2XRWYHDFUBQGQ6BGI6ST5VQULEFM4WMIDA, seg_pos={0 0}, upload=440596]: Done uploading piece. err: <nil> Dec 26 13:31:04.975722420 (502h2m4.846805455s) [noise=true , address=145.79.227.46:4601 , piece_id=7LMVQBMEV4HAC55YX677QP6YYRRXRBJ6JMQEZMPVBOP76DKNASRA, seg_pos={0 0}, upload=440596]: Done uploading piece. err: <nil> Dec 26 13:31:04.980280069 (502h2m4.851363114s) [noise=true , address=145.79.227.13:4804 , piece_id=VR4HBYAA2URAYZ4PE3GZBP4LKDIPPY5GBSVBY4ZHC6CDWCJEAOUQ, seg_pos={0 0}, upload=440596]: Done uploading piece. err: <nil> Dec 26 13:31:04.983441719 (502h2m4.854524764s) [noise=true , address=145.79.227.42:4121 , piece_id=QMDX2HM25FEYRGSZEJD3ER25QEG4ILL7M6MT4OMUQNH564P6NWYQ, seg_pos={0 0}, upload=440596]: Done uploading piece. err: <nil> Dec 26 13:31:04.984230942 (502h2m4.855313987s) [noise=true , address=145.79.227.17:4970 , piece_id=SCM4BGWZDM32LWDRWUY5OFVMXO27DEXLGSGOR42RGNGSTQZ6ZHIQ, seg_pos={0 0}, upload=440596]: Done uploading piece. err: <nil> Dec 26 13:31:05.000047099 (502h2m4.871130154s) [noise=true , address=145.79.227.16:4910 , piece_id=FQCIMCSMA76G5ATBMFKXOUEF2AV75UOT6H6FGPTQTP6YCK43KOPQ, seg_pos={0 0}, upload=440596]: Done uploading piece. err: <nil> Dec 26 13:31:05.000415691 (502h2m4.871498736s) [noise=true , address=145.79.227.45:4230 , piece_id=7JFEWPGLA2URVDPDXZZM3GY4JFAXO3VENBZ3ACVS4C2XZQA2KIOQ, seg_pos={0 0}, upload=440596]: Done uploading piece. err: <nil> Dec 26 13:31:05.004974081 (502h2m4.876057126s) [noise=true , address=145.79.227.26:4654 , piece_id=YZTKQ3L2Z22GDXF4SJ224AS4IEDFPPVHHD5QKCDOT7ZIFWDVSA6Q, seg_pos={0 0}, upload=440596]: Done uploading piece. err: <nil> Dec 26 13:31:05.211910567 (502h2m5.082993612s) [noise=true , address=45.91.116.12:4225 , piece_id=GEHXFI6R66VTL4J22OUJEBURZHVA6T6GBYRWVBDSELZDP5DFNKNQ, seg_pos={0 0}, upload=440596]: Done uploading piece. err: <nil> Dec 26 13:31:05.257675882 (502h2m5.128758917s) [noise=true , address=45.91.116.8:4047 , piece_id=ZCL6DLT2KZTY2U67GF3Q7NV5WRVM7AFWRMH4PEW3XGJTLA522TOQ, seg_pos={0 0}, upload=440596]: Done uploading piece. err: <nil> Dec 26 13:31:05.277732114 (502h2m5.148815159s) [noise=true , address=77.90.31.21:4878 , piece_id=C2P247WP3OZMADOYB4WFN4BPQICO2PQVDZ7FE64TZ6HH7ONE7WVQ, seg_pos={0 0}, upload=440596]: Done uploading piece. err: <nil> Dec 26 13:31:05.277755327 (502h2m5.148838362s) [seg_pos={0 0}, upload=440596]: Segment reached optimal threshold of 13 pieces. Dec 26 13:31:05.278105464 (502h2m5.149188499s) [noise=true , address=45.91.116.11:4120 , piece_id=UIZ7STB3DVXXHEJ6VE5DLUKDHBCQ37SIQO5EQHHXWMXAOWOVL2JA, seg_pos={0 0}, upload=440596]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 26 13:31:05.278114121 (502h2m5.149197156s) [noise=true , address=45.91.116.11:4120 , piece_id=UIZ7STB3DVXXHEJ6VE5DLUKDHBCQ37SIQO5EQHHXWMXAOWOVL2JA, seg_pos={0 0}, upload=440596]: Long tail cancellation due to optimal threshold Dec 26 13:31:05.278184876 (502h2m5.149267921s) [noise=true , address=45.91.116.13:4154 , piece_id=X3PGD4LFI2QXAPXII3P3YM2AQRVDFLN4FP2HK245OECTCQ6RAAUQ, seg_pos={0 0}, upload=440596]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 26 13:31:05.278206066 (502h2m5.149289101s) [noise=true , address=45.91.116.13:4154 , piece_id=X3PGD4LFI2QXAPXII3P3YM2AQRVDFLN4FP2HK245OECTCQ6RAAUQ, seg_pos={0 0}, upload=440596]: Long tail cancellation due to optimal threshold Dec 26 13:31:05.278409824 (502h2m5.149492899s) [noise=true , address=77.90.31.15:4700 , piece_id=BEVWL5ON5FOZQLQAZSEINWBMKDWDE7KRPN66GFSK2M5AXDACFMJA, seg_pos={0 0}, upload=440596]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: read tcp 188.166.225.170:40762->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 13:31:05.278425444 (502h2m5.149508479s) [noise=true , address=77.90.31.15:4700 , piece_id=BEVWL5ON5FOZQLQAZSEINWBMKDWDE7KRPN66GFSK2M5AXDACFMJA, seg_pos={0 0}, upload=440596]: Long tail cancellation due to optimal threshold Dec 26 13:31:05.278449289 (502h2m5.149532324s) [seg_pos={0 0}, upload=440596]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 13:31:05.278523600 (502h2m5.149606635s) [upload=440596]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:31:05.278528821 (502h2m5.149611866s) [upload=440596]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:31:05.565379729 (502h2m5.436462774s) [upload=440596]: Done uploading segments... Dec 26 13:31:05.565408474 (502h2m5.436491519s) [upload=440596]: Upload finished. err: <nil> Duration: 30.005405934s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 16 out of 60 is failed (threshold=3) [....X.XXX...X..X...................XX.X........X.XXX.X.X..X.] duration: 42.013468534s trace: https://jaeger.dev.storj.io/trace/62a5e8136837c4e FAIL (2025-12-26 13:31:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12N8gL2cPKtuvwn66hcZWQsoe3KRUYgHoy5FTJ38KZstPzzxiKt, Piece ID: GIU2CB4NSOCHLR4XBBB53GNZ4BYZUGDVZVPEPXA6DJQ5MH7EBIBA) context deadline exceeded; piecestore: (Node ID: 1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, Piece ID: 2WQIOK5KRROZZOY63DDCT62IH5FQQMWKJZIX3YRRUJ6KCZ76THHQ) context deadline exceeded; piecestore: (Node ID: 126oNZ8ZQpENsHFQy2RCab3g9eNJ1JBVqxNkcB7mMCXTftap84W, Piece ID: TE3G3HW7P2JM3DUTBI44USPC3IWJMFZS3BY6JDFDZN2J4SMC4IHA) context deadline exceeded ======LOGS====== 26 13:29:32.278564562 (502h0m32.149647627s) [noise=true , address=145.79.227.30:4796 , piece_id=ITAXIGGPL3V7GFENXZHTI3P3UCVWKP6HP3RI2NHBHL7HCN2GR2JQ, seg_pos={0 0}, upload=440573]: Uploading piece... Dec 26 13:29:32.279229668 (502h0m32.150312703s) [noise=true , address=77.90.31.17:4776 , piece_id=4WY3HFUKERU6GJ6WWPEKUYI7DCLTYN5TXG6AHIJCBIMMYHYPCKRQ, seg_pos={0 0}, upload=440573]: Uploading piece... Dec 26 13:29:32.279784835 (502h0m32.15086787s) [noise=true , address=77.90.31.19:4869 , piece_id=C2IYXUPGEBA7V7FOEIPOAQHVCZ5BICC7OS6DV6UBSGJXMMVDVRLA, seg_pos={0 0}, upload=440573]: Uploading piece... Dec 26 13:29:32.280284757 (502h0m32.151367792s) [noise=true , address=145.79.227.40:4067 , piece_id=C5V5JGRNMF2AXAIYP5KBIEVTJFRK6R72A2X2EYQRYQH345EIYCMQ, seg_pos={0 0}, upload=440573]: Uploading piece... Dec 26 13:29:32.280751957 (502h0m32.151835002s) [noise=true , address=45.91.116.7:4042 , piece_id=RQZSXDO6TETWGCIMF6CCTFETMOODBVIFIYXHZKDNSVW5BYBRZX4A, seg_pos={0 0}, upload=440573]: Uploading piece... Dec 26 13:29:32.281195110 (502h0m32.152278145s) [noise=true , address=77.90.31.14:4671 , piece_id=TE3G3HW7P2JM3DUTBI44USPC3IWJMFZS3BY6JDFDZN2J4SMC4IHA, seg_pos={0 0}, upload=440573]: Uploading piece... Dec 26 13:29:32.281909341 (502h0m32.152992386s) [noise=true , address=145.79.227.44:4205 , piece_id=5OSOHHKSVD3LKXPZ3HJFCWY6ZKWSTNCIKNFZTKVBME4CFZAHL3GA, seg_pos={0 0}, upload=440573]: Uploading piece... Dec 26 13:29:32.281979484 (502h0m32.153062519s) [noise=true , address=77.90.31.15:4706 , piece_id=WPMHDF2AKEPXW5CN34DFH7T7PCQ4PRQUM5G6EFMQAS2HW3ITWPLA, seg_pos={0 0}, upload=440573]: Uploading piece... Dec 26 13:29:32.282653828 (502h0m32.153736863s) [noise=true , address=145.79.227.48:4329 , piece_id=GLJAQM5YP3LCCSGHOUMQARORY6ZIGNIMJQPVFW56HHSHQKU3O64Q, seg_pos={0 0}, upload=440573]: Uploading piece... Dec 26 13:29:32.282881913 (502h0m32.153964958s) [noise=true , address=145.79.227.52:4449 , piece_id=NZNYADQDANYMUQPZ26UFDXAB5CQF6ZZBQR3JCUR22FFA3WYH5FSA, seg_pos={0 0}, upload=440573]: Uploading piece... Dec 26 13:29:32.283155243 (502h0m32.154238278s) [noise=true , address=45.91.116.14:4195 , piece_id=GIU2CB4NSOCHLR4XBBB53GNZ4BYZUGDVZVPEPXA6DJQ5MH7EBIBA, seg_pos={0 0}, upload=440573]: Uploading piece... Dec 26 13:29:32.283420818 (502h0m32.154503863s) [noise=true , address=64.49.13.169:4706 , piece_id=ZHYAEU6IOT5FIRXRBNI2LMSWGVCORFL2MLUVEUTZDCPWYJ3NUETQ, seg_pos={0 0}, upload=440573]: Uploading piece... Dec 26 13:29:32.283731360 (502h0m32.154814395s) [noise=true , address=77.90.31.18:4810 , piece_id=R37QEJSHY47UUCI4IFQCVHEDUCE6AN6WU3VRNR67KXTL7EJ5VCJA, seg_pos={0 0}, upload=440573]: Uploading piece... Dec 26 13:29:32.284141471 (502h0m32.155224516s) [noise=true , address=77.90.31.11:4569 , piece_id=VHWTWGDHTLAJFEELYM5XNXOHXGFFDG2EBB2JXHAUUC57FLGQNQEA, seg_pos={0 0}, upload=440573]: Uploading piece... Dec 26 13:29:32.284487740 (502h0m32.155570775s) [noise=true , address=45.91.116.12:4235 , piece_id=2WQIOK5KRROZZOY63DDCT62IH5FQQMWKJZIX3YRRUJ6KCZ76THHQ, seg_pos={0 0}, upload=440573]: Uploading piece... Dec 26 13:29:32.284814412 (502h0m32.155897447s) [noise=true , address=77.90.31.16:4763 , piece_id=RPFTQSBCHPRKCO3WUXMZQP4LDWZB6HF6X3OL4Y5VHTXXUGLUC5HA, seg_pos={0 0}, upload=440573]: Uploading piece... Dec 26 13:29:33.755237461 (502h0m33.626320496s) [upload=440573]: Next returned nil segment Dec 26 13:29:33.755251126 (502h0m33.626334161s) [upload=440573]: Waiting for error group managing segments... Dec 26 13:29:34.167217093 (502h0m34.038300138s) [noise=true , address=145.79.227.40:4067 , piece_id=C5V5JGRNMF2AXAIYP5KBIEVTJFRK6R72A2X2EYQRYQH345EIYCMQ, seg_pos={0 0}, upload=440573]: Done uploading piece. err: <nil> Dec 26 13:29:34.173677816 (502h0m34.044760861s) [noise=true , address=145.79.227.52:4449 , piece_id=NZNYADQDANYMUQPZ26UFDXAB5CQF6ZZBQR3JCUR22FFA3WYH5FSA, seg_pos={0 0}, upload=440573]: Done uploading piece. err: <nil> Dec 26 13:29:34.174544977 (502h0m34.045628012s) [noise=true , address=145.79.227.48:4329 , piece_id=GLJAQM5YP3LCCSGHOUMQARORY6ZIGNIMJQPVFW56HHSHQKU3O64Q, seg_pos={0 0}, upload=440573]: Done uploading piece. err: <nil> Dec 26 13:29:34.193585975 (502h0m34.06466902s) [noise=true , address=145.79.227.29:4970 , piece_id=CMQTNSCP4DSIABKSNXYV4KSPRIVYBQ4YPU4ENLRVOXSUNIETQD6A, seg_pos={0 0}, upload=440573]: Done uploading piece. err: <nil> Dec 26 13:29:34.323466150 (502h0m34.194549185s) [noise=true , address=77.90.31.18:4810 , piece_id=R37QEJSHY47UUCI4IFQCVHEDUCE6AN6WU3VRNR67KXTL7EJ5VCJA, seg_pos={0 0}, upload=440573]: Done uploading piece. err: <nil> Dec 26 13:29:34.392077067 (502h0m34.263160102s) [noise=true , address=45.91.116.14:4195 , piece_id=GIU2CB4NSOCHLR4XBBB53GNZ4BYZUGDVZVPEPXA6DJQ5MH7EBIBA, seg_pos={0 0}, upload=440573]: Done uploading piece. err: <nil> Dec 26 13:29:34.439840176 (502h0m34.310923211s) [noise=true , address=77.90.31.19:4869 , piece_id=C2IYXUPGEBA7V7FOEIPOAQHVCZ5BICC7OS6DV6UBSGJXMMVDVRLA, seg_pos={0 0}, upload=440573]: Done uploading piece. err: <nil> Dec 26 13:29:34.486767434 (502h0m34.357850469s) [noise=true , address=77.90.31.15:4706 , piece_id=WPMHDF2AKEPXW5CN34DFH7T7PCQ4PRQUM5G6EFMQAS2HW3ITWPLA, seg_pos={0 0}, upload=440573]: Done uploading piece. err: <nil> Dec 26 13:29:34.507965730 (502h0m34.379048775s) [noise=true , address=77.90.31.11:4569 , piece_id=VHWTWGDHTLAJFEELYM5XNXOHXGFFDG2EBB2JXHAUUC57FLGQNQEA, seg_pos={0 0}, upload=440573]: Done uploading piece. err: <nil> Dec 26 13:29:34.528676206 (502h0m34.399759241s) [noise=true , address=45.91.116.7:4042 , piece_id=RQZSXDO6TETWGCIMF6CCTFETMOODBVIFIYXHZKDNSVW5BYBRZX4A, seg_pos={0 0}, upload=440573]: Done uploading piece. err: <nil> Dec 26 13:29:34.554686674 (502h0m34.42576971s) [noise=true , address=77.90.31.16:4763 , piece_id=RPFTQSBCHPRKCO3WUXMZQP4LDWZB6HF6X3OL4Y5VHTXXUGLUC5HA, seg_pos={0 0}, upload=440573]: Done uploading piece. err: <nil> Dec 26 13:29:34.817696482 (502h0m34.688779517s) [noise=true , address=77.90.31.12:4602 , piece_id=FLGZ4U6PWZNKH3FAS2YG3IAAA5XGMIBLBEYLCJ2PVGZDG6YBRD6Q, seg_pos={0 0}, upload=440573]: Done uploading piece. err: <nil> Dec 26 13:29:34.899012282 (502h0m34.770095328s) [noise=true , address=77.90.31.14:4671 , piece_id=TE3G3HW7P2JM3DUTBI44USPC3IWJMFZS3BY6JDFDZN2J4SMC4IHA, seg_pos={0 0}, upload=440573]: Done uploading piece. err: <nil> Dec 26 13:29:35.059110307 (502h0m34.930193352s) [noise=true , address=45.91.116.12:4235 , piece_id=2WQIOK5KRROZZOY63DDCT62IH5FQQMWKJZIX3YRRUJ6KCZ76THHQ, seg_pos={0 0}, upload=440573]: Done uploading piece. err: <nil> Dec 26 13:29:35.059135465 (502h0m34.93021851s) [seg_pos={0 0}, upload=440573]: Segment reached optimal threshold of 14 pieces. Dec 26 13:29:35.059365774 (502h0m34.930448809s) [noise=true , address=77.90.31.17:4776 , piece_id=4WY3HFUKERU6GJ6WWPEKUYI7DCLTYN5TXG6AHIJCBIMMYHYPCKRQ, seg_pos={0 0}, upload=440573]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Dec 26 13:29:35.059374851 (502h0m34.930457886s) [noise=true , address=77.90.31.17:4776 , piece_id=4WY3HFUKERU6GJ6WWPEKUYI7DCLTYN5TXG6AHIJCBIMMYHYPCKRQ, seg_pos={0 0}, upload=440573]: Long tail cancellation due to optimal threshold Dec 26 13:29:35.059671406 (502h0m34.930754451s) [noise=true , address=145.79.227.44:4205 , piece_id=5OSOHHKSVD3LKXPZ3HJFCWY6ZKWSTNCIKNFZTKVBME4CFZAHL3GA, seg_pos={0 0}, upload=440573]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UafxBVo9nuRVBcQmGE4gBP2pnXQ5YUrwHZdwPCeSBSBWCZKpQ): protocol: expected piece hash; context canceled; EOF Dec 26 13:29:35.059679611 (502h0m34.930762656s) [noise=true , address=145.79.227.44:4205 , piece_id=5OSOHHKSVD3LKXPZ3HJFCWY6ZKWSTNCIKNFZTKVBME4CFZAHL3GA, seg_pos={0 0}, upload=440573]: Long tail cancellation due to optimal threshold Dec 26 13:29:35.059826621 (502h0m34.930909656s) [noise=true , address=64.49.13.169:4706 , piece_id=ZHYAEU6IOT5FIRXRBNI2LMSWGVCORFL2MLUVEUTZDCPWYJ3NUETQ, seg_pos={0 0}, upload=440573]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:47026->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:47026->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47026->64.49.13.169:4706: use of closed network connection Dec 26 13:29:35.059834075 (502h0m34.93091713s) [noise=true , address=64.49.13.169:4706 , piece_id=ZHYAEU6IOT5FIRXRBNI2LMSWGVCORFL2MLUVEUTZDCPWYJ3NUETQ, seg_pos={0 0}, upload=440573]: Long tail cancellation due to optimal threshold Dec 26 13:29:35.060007676 (502h0m34.931090721s) [noise=true , address=145.79.227.30:4796 , piece_id=ITAXIGGPL3V7GFENXZHTI3P3UCVWKP6HP3RI2NHBHL7HCN2GR2JQ, seg_pos={0 0}, upload=440573]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 26 13:29:35.060016282 (502h0m34.931099317s) [noise=true , address=145.79.227.30:4796 , piece_id=ITAXIGGPL3V7GFENXZHTI3P3UCVWKP6HP3RI2NHBHL7HCN2GR2JQ, seg_pos={0 0}, upload=440573]: Long tail cancellation due to optimal threshold Dec 26 13:29:35.060036901 (502h0m34.931119947s) [seg_pos={0 0}, upload=440573]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 13:29:35.060109039 (502h0m34.931192084s) [upload=440573]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:29:35.060116573 (502h0m34.931199618s) [upload=440573]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:29:35.362502901 (502h0m35.233585936s) [upload=440573]: Done uploading segments... Dec 26 13:29:35.362548688 (502h0m35.233631733s) [upload=440573]: Upload finished. err: <nil> Duration: 42.013498329s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 32 out of 60 is failed (threshold=6) [XX.XXXX.X..X..X....XXXXXX...XX.....X...XX.X.XXX.XXXXX..X.X.X] duration: 5.002201505s trace: https://jaeger.dev.storj.io/trace/c1b9573f2752f26 FAIL (2025-12-26 13:31:07 UTC) orbiter warning: context deadline exceeded Duration: 5.002246782s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 32 out of 60 is failed (threshold=6) [XXX.XXXX.X..X..X....XXXXXX...XX.....X...XX.X.XXX.XXXXX..X.X.] duration: 3.846906361s trace: https://jaeger.dev.storj.io/trace/7db04fcc87596bb8 FAIL (2025-12-26 13:30:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.846936006s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X..XX.....X..XXXXXX..X....X..X.X....X......XX.X.X......X.X.] duration: 6.695672777s trace: https://jaeger.dev.storj.io/trace/1c0b85a1d6706345 FAIL (2025-12-26 13:30:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: R7DFCLKBBX27VDKRZXNFPM6WJKBPDHBZFTZMZZB7TJ2ISGQNCM6Q) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: N3HI2LUWCK642HS6WB46NBHMJMFAHW3LH6OGSOK56CQRIUV2W5MA) context deadline exceeded ======LOGS====== d Dec 26 13:29:59.128734722 (502h0m58.999817757s) [upload=440577]: Uploading segments... Dec 26 13:29:59.128766392 (502h0m58.999849427s) [upload=440577]: Got next segment. Inline: false Dec 26 13:29:59.128792242 (502h0m58.999875277s) [upload=440577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 13:29:59.128796550 (502h0m58.999879585s) [upload=440577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 13:29:59.373533585 (502h0m59.24461662s) [seg_pos={0 0}, upload=440577]: Begin upload segment... Dec 26 13:29:59.373579592 (502h0m59.244662627s) [seg_pos={0 0}, upload=440577]: Done begin upload segment. Dec 26 13:29:59.373604510 (502h0m59.244687545s) [noise=true , address=145.79.227.35:4483 , piece_id=5TFECXO62MBXQDCTL7XH5QX7AMRKDMSWNEWQXQ6LDCA4HJRLP4UQ, seg_pos={0 0}, upload=440577]: Uploading piece... Dec 26 13:29:59.373635088 (502h0m59.244718123s) [noise=true , address=45.91.116.11:4118 , piece_id=WTOL5RDDLUTWVPCZNHYESH3FOVKF2ROP62H4B7LAHKWGBGOAMMUA, seg_pos={0 0}, upload=440577]: Uploading piece... Dec 26 13:29:59.374131082 (502h0m59.245214117s) [noise=true , address=145.79.227.17:4970 , piece_id=WFMPDUQA6VB67Q4KFT7G6DOENDKF5AGNTNUQ35AKBBZ4HKFY5RRQ, seg_pos={0 0}, upload=440577]: Uploading piece... Dec 26 13:29:59.374410124 (502h0m59.245493159s) [noise=true , address=145.79.227.37:4567 , piece_id=IYZD7BEHY4GHAX54MTHL7ETT4SRREIAXKYYVXUYNY6VZLYPF27VQ, seg_pos={0 0}, upload=440577]: Uploading piece... Dec 26 13:29:59.374710787 (502h0m59.245793822s) [noise=true , address=145.79.227.16:4910 , piece_id=6QBNN3C36DS2W7JGDGW235ZDEQQFNPTLR4WQDSXB4JHSJ5I67PGQ, seg_pos={0 0}, upload=440577]: Uploading piece... Dec 26 13:29:59.374987624 (502h0m59.246070659s) [noise=true , address=45.91.116.10:4082 , piece_id=7GQEWAVI6XPBC2CBNCSUXG76HWM5CY53H4JHNBGBKDEL4C7MK7OA, seg_pos={0 0}, upload=440577]: Uploading piece... Dec 26 13:29:59.375272767 (502h0m59.246355802s) [noise=true , address=45.91.116.5:4300 , piece_id=LIQNDNL2VMBKUKN5LOQEJMAEUDYPPWWPAQIFV74QTIHACLRGVRHQ, seg_pos={0 0}, upload=440577]: Uploading piece... Dec 26 13:29:59.375520398 (502h0m59.246603433s) [noise=true , address=77.90.31.14:4663 , piece_id=N3HI2LUWCK642HS6WB46NBHMJMFAHW3LH6OGSOK56CQRIUV2W5MA, seg_pos={0 0}, upload=440577]: Uploading piece... Dec 26 13:29:59.375811041 (502h0m59.246894086s) [noise=true , address=145.79.227.4:4186 , piece_id=7PKODQ5C3RCWU62OAMFVHNVTDNWZM6HLPSXAWPERGRYNIWHTDXUQ, seg_pos={0 0}, upload=440577]: Uploading piece... Dec 26 13:29:59.376094110 (502h0m59.247177145s) [noise=true , address=145.79.227.48:4302 , piece_id=YCYMVKWFYCQGAML7BXUUPTPSNB7E4LWDOQX46ABEWZIFHWWPLIUA, seg_pos={0 0}, upload=440577]: Uploading piece... Dec 26 13:29:59.377954223 (502h0m59.249037268s) [noise=true , address=145.79.227.29:4941 , piece_id=6IFO337QZWSREFCZAH35ULLVILC6SG4V4VXKRZDVYHXUJUDGWY6A, seg_pos={0 0}, upload=440577]: Uploading piece... Dec 26 13:29:59.378312414 (502h0m59.249395449s) [noise=true , address=77.90.31.17:4773 , piece_id=KQI4XXGDAXC6IKX25L5MUXZCH4TV5YCXTCWKLCBLJVOJ2C72RE4A, seg_pos={0 0}, upload=440577]: Uploading piece... Dec 26 13:29:59.378659445 (502h0m59.24974248s) [noise=true , address=77.90.31.16:4735 , piece_id=R7DFCLKBBX27VDKRZXNFPM6WJKBPDHBZFTZMZZB7TJ2ISGQNCM6Q, seg_pos={0 0}, upload=440577]: Uploading piece... Dec 26 13:29:59.379690689 (502h0m59.250773724s) [noise=true , address=45.91.116.4:4939 , piece_id=BPMADH3DHW5GXDNE5KTAU2RJZTE3D3DVTJJWN73M3QKB73FMRWDA, seg_pos={0 0}, upload=440577]: Uploading piece... Dec 26 13:29:59.380824437 (502h0m59.251907473s) [noise=true , address=145.79.227.22:4511 , piece_id=S4TQXZ2PQAECOEUCRA735TSQCHMEJENKSCX3BLXN7RLFD6P5E6HQ, seg_pos={0 0}, upload=440577]: Uploading piece... Dec 26 13:29:59.382145864 (502h0m59.253228899s) [noise=true , address=45.91.116.9:4261 , piece_id=JUJEWFFPAVRWKUSA47NDQBIAP7DO6DWVCMXPIVZWBIPTWS5W4DXQ, seg_pos={0 0}, upload=440577]: Uploading piece... Dec 26 13:30:01.100561450 (502h1m0.971644485s) [upload=440577]: Next returned nil segment Dec 26 13:30:01.100583282 (502h1m0.971666317s) [upload=440577]: Waiting for error group managing segments... Dec 26 13:30:01.377491026 (502h1m1.248574071s) [noise=true , address=145.79.227.48:4302 , piece_id=YCYMVKWFYCQGAML7BXUUPTPSNB7E4LWDOQX46ABEWZIFHWWPLIUA, seg_pos={0 0}, upload=440577]: Done uploading piece. err: <nil> Dec 26 13:30:01.388085129 (502h1m1.259168174s) [noise=true , address=145.79.227.29:4941 , piece_id=6IFO337QZWSREFCZAH35ULLVILC6SG4V4VXKRZDVYHXUJUDGWY6A, seg_pos={0 0}, upload=440577]: Done uploading piece. err: <nil> Dec 26 13:30:01.407082083 (502h1m1.278165128s) [noise=true , address=145.79.227.37:4567 , piece_id=IYZD7BEHY4GHAX54MTHL7ETT4SRREIAXKYYVXUYNY6VZLYPF27VQ, seg_pos={0 0}, upload=440577]: Done uploading piece. err: <nil> Dec 26 13:30:01.408405583 (502h1m1.279488638s) [noise=true , address=145.79.227.17:4970 , piece_id=WFMPDUQA6VB67Q4KFT7G6DOENDKF5AGNTNUQ35AKBBZ4HKFY5RRQ, seg_pos={0 0}, upload=440577]: Done uploading piece. err: <nil> Dec 26 13:30:01.411481089 (502h1m1.282564134s) [noise=true , address=145.79.227.16:4910 , piece_id=6QBNN3C36DS2W7JGDGW235ZDEQQFNPTLR4WQDSXB4JHSJ5I67PGQ, seg_pos={0 0}, upload=440577]: Done uploading piece. err: <nil> Dec 26 13:30:01.412343121 (502h1m1.283426166s) [noise=true , address=145.79.227.22:4511 , piece_id=S4TQXZ2PQAECOEUCRA735TSQCHMEJENKSCX3BLXN7RLFD6P5E6HQ, seg_pos={0 0}, upload=440577]: Done uploading piece. err: <nil> Dec 26 13:30:01.414364319 (502h1m1.285447364s) [noise=true , address=145.79.227.35:4483 , piece_id=5TFECXO62MBXQDCTL7XH5QX7AMRKDMSWNEWQXQ6LDCA4HJRLP4UQ, seg_pos={0 0}, upload=440577]: Done uploading piece. err: <nil> Dec 26 13:30:01.421545965 (502h1m1.292629s) [noise=true , address=145.79.227.4:4186 , piece_id=7PKODQ5C3RCWU62OAMFVHNVTDNWZM6HLPSXAWPERGRYNIWHTDXUQ, seg_pos={0 0}, upload=440577]: Done uploading piece. err: <nil> Dec 26 13:30:01.599616284 (502h1m1.470699329s) [noise=true , address=77.90.31.14:4663 , piece_id=N3HI2LUWCK642HS6WB46NBHMJMFAHW3LH6OGSOK56CQRIUV2W5MA, seg_pos={0 0}, upload=440577]: Done uploading piece. err: <nil> Dec 26 13:30:01.612627741 (502h1m1.483710786s) [noise=true , address=77.90.31.16:4735 , piece_id=R7DFCLKBBX27VDKRZXNFPM6WJKBPDHBZFTZMZZB7TJ2ISGQNCM6Q, seg_pos={0 0}, upload=440577]: Done uploading piece. err: <nil> Dec 26 13:30:01.706133695 (502h1m1.57721675s) [noise=true , address=45.91.116.4:4939 , piece_id=BPMADH3DHW5GXDNE5KTAU2RJZTE3D3DVTJJWN73M3QKB73FMRWDA, seg_pos={0 0}, upload=440577]: Done uploading piece. err: <nil> Dec 26 13:30:01.768761910 (502h1m1.639844965s) [noise=true , address=77.90.31.17:4773 , piece_id=KQI4XXGDAXC6IKX25L5MUXZCH4TV5YCXTCWKLCBLJVOJ2C72RE4A, seg_pos={0 0}, upload=440577]: Done uploading piece. err: <nil> Dec 26 13:30:01.819194072 (502h1m1.690277117s) [noise=true , address=45.91.116.11:4118 , piece_id=WTOL5RDDLUTWVPCZNHYESH3FOVKF2ROP62H4B7LAHKWGBGOAMMUA, seg_pos={0 0}, upload=440577]: Done uploading piece. err: <nil> Dec 26 13:30:01.819232816 (502h1m1.690315861s) [seg_pos={0 0}, upload=440577]: Segment reached optimal threshold of 13 pieces. Dec 26 13:30:01.819483684 (502h1m1.690566739s) [noise=true , address=45.91.116.9:4261 , piece_id=JUJEWFFPAVRWKUSA47NDQBIAP7DO6DWVCMXPIVZWBIPTWS5W4DXQ, seg_pos={0 0}, upload=440577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 26 13:30:01.819494404 (502h1m1.690577449s) [noise=true , address=45.91.116.9:4261 , piece_id=JUJEWFFPAVRWKUSA47NDQBIAP7DO6DWVCMXPIVZWBIPTWS5W4DXQ, seg_pos={0 0}, upload=440577]: Long tail cancellation due to optimal threshold Dec 26 13:30:01.819949741 (502h1m1.691032786s) [noise=true , address=45.91.116.10:4082 , piece_id=7GQEWAVI6XPBC2CBNCSUXG76HWM5CY53H4JHNBGBKDEL4C7MK7OA, seg_pos={0 0}, upload=440577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:37612->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:37612->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37612->45.91.116.10:4082: use of closed network connection Dec 26 13:30:01.819962285 (502h1m1.69104533s) [noise=true , address=45.91.116.10:4082 , piece_id=7GQEWAVI6XPBC2CBNCSUXG76HWM5CY53H4JHNBGBKDEL4C7MK7OA, seg_pos={0 0}, upload=440577]: Long tail cancellation due to optimal threshold Dec 26 13:30:01.820110167 (502h1m1.691193212s) [noise=true , address=45.91.116.5:4300 , piece_id=LIQNDNL2VMBKUKN5LOQEJMAEUDYPPWWPAQIFV74QTIHACLRGVRHQ, seg_pos={0 0}, upload=440577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:37800->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:37800->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37800->45.91.116.5:4300: use of closed network connection Dec 26 13:30:01.820120165 (502h1m1.691203211s) [noise=true , address=45.91.116.5:4300 , piece_id=LIQNDNL2VMBKUKN5LOQEJMAEUDYPPWWPAQIFV74QTIHACLRGVRHQ, seg_pos={0 0}, upload=440577]: Long tail cancellation due to optimal threshold Dec 26 13:30:01.820141887 (502h1m1.691224932s) [seg_pos={0 0}, upload=440577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 13:30:01.820237539 (502h1m1.691320584s) [upload=440577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:30:01.820245745 (502h1m1.69132879s) [upload=440577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:30:02.111713364 (502h1m1.982796409s) [upload=440577]: Done uploading segments... Dec 26 13:30:02.111742239 (502h1m1.982825284s) [upload=440577]: Upload finished. err: <nil> Duration: 6.695691713s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 16 out of 60 is failed (threshold=3) [.....X.XXX...X..X...................XX.X........X.XXX.X.X..X] duration: 1m0.003596007s trace: https://jaeger.dev.storj.io/trace/62a5e8136837c4e FAIL (2025-12-26 13:30:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12N8gL2cPKtuvwn66hcZWQsoe3KRUYgHoy5FTJ38KZstPzzxiKt, Piece ID: GIU2CB4NSOCHLR4XBBB53GNZ4BYZUGDVZVPEPXA6DJQ5MH7EBIBA) context deadline exceeded; piecestore: (Node ID: 1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, Piece ID: 2WQIOK5KRROZZOY63DDCT62IH5FQQMWKJZIX3YRRUJ6KCZ76THHQ) context deadline exceeded; piecestore: (Node ID: 126oNZ8ZQpENsHFQy2RCab3g9eNJ1JBVqxNkcB7mMCXTftap84W, Piece ID: TE3G3HW7P2JM3DUTBI44USPC3IWJMFZS3BY6JDFDZN2J4SMC4IHA) context deadline exceeded ======LOGS====== 26 13:29:32.278564562 (502h0m32.149647627s) [noise=true , address=145.79.227.30:4796 , piece_id=ITAXIGGPL3V7GFENXZHTI3P3UCVWKP6HP3RI2NHBHL7HCN2GR2JQ, seg_pos={0 0}, upload=440573]: Uploading piece... Dec 26 13:29:32.279229668 (502h0m32.150312703s) [noise=true , address=77.90.31.17:4776 , piece_id=4WY3HFUKERU6GJ6WWPEKUYI7DCLTYN5TXG6AHIJCBIMMYHYPCKRQ, seg_pos={0 0}, upload=440573]: Uploading piece... Dec 26 13:29:32.279784835 (502h0m32.15086787s) [noise=true , address=77.90.31.19:4869 , piece_id=C2IYXUPGEBA7V7FOEIPOAQHVCZ5BICC7OS6DV6UBSGJXMMVDVRLA, seg_pos={0 0}, upload=440573]: Uploading piece... Dec 26 13:29:32.280284757 (502h0m32.151367792s) [noise=true , address=145.79.227.40:4067 , piece_id=C5V5JGRNMF2AXAIYP5KBIEVTJFRK6R72A2X2EYQRYQH345EIYCMQ, seg_pos={0 0}, upload=440573]: Uploading piece... Dec 26 13:29:32.280751957 (502h0m32.151835002s) [noise=true , address=45.91.116.7:4042 , piece_id=RQZSXDO6TETWGCIMF6CCTFETMOODBVIFIYXHZKDNSVW5BYBRZX4A, seg_pos={0 0}, upload=440573]: Uploading piece... Dec 26 13:29:32.281195110 (502h0m32.152278145s) [noise=true , address=77.90.31.14:4671 , piece_id=TE3G3HW7P2JM3DUTBI44USPC3IWJMFZS3BY6JDFDZN2J4SMC4IHA, seg_pos={0 0}, upload=440573]: Uploading piece... Dec 26 13:29:32.281909341 (502h0m32.152992386s) [noise=true , address=145.79.227.44:4205 , piece_id=5OSOHHKSVD3LKXPZ3HJFCWY6ZKWSTNCIKNFZTKVBME4CFZAHL3GA, seg_pos={0 0}, upload=440573]: Uploading piece... Dec 26 13:29:32.281979484 (502h0m32.153062519s) [noise=true , address=77.90.31.15:4706 , piece_id=WPMHDF2AKEPXW5CN34DFH7T7PCQ4PRQUM5G6EFMQAS2HW3ITWPLA, seg_pos={0 0}, upload=440573]: Uploading piece... Dec 26 13:29:32.282653828 (502h0m32.153736863s) [noise=true , address=145.79.227.48:4329 , piece_id=GLJAQM5YP3LCCSGHOUMQARORY6ZIGNIMJQPVFW56HHSHQKU3O64Q, seg_pos={0 0}, upload=440573]: Uploading piece... Dec 26 13:29:32.282881913 (502h0m32.153964958s) [noise=true , address=145.79.227.52:4449 , piece_id=NZNYADQDANYMUQPZ26UFDXAB5CQF6ZZBQR3JCUR22FFA3WYH5FSA, seg_pos={0 0}, upload=440573]: Uploading piece... Dec 26 13:29:32.283155243 (502h0m32.154238278s) [noise=true , address=45.91.116.14:4195 , piece_id=GIU2CB4NSOCHLR4XBBB53GNZ4BYZUGDVZVPEPXA6DJQ5MH7EBIBA, seg_pos={0 0}, upload=440573]: Uploading piece... Dec 26 13:29:32.283420818 (502h0m32.154503863s) [noise=true , address=64.49.13.169:4706 , piece_id=ZHYAEU6IOT5FIRXRBNI2LMSWGVCORFL2MLUVEUTZDCPWYJ3NUETQ, seg_pos={0 0}, upload=440573]: Uploading piece... Dec 26 13:29:32.283731360 (502h0m32.154814395s) [noise=true , address=77.90.31.18:4810 , piece_id=R37QEJSHY47UUCI4IFQCVHEDUCE6AN6WU3VRNR67KXTL7EJ5VCJA, seg_pos={0 0}, upload=440573]: Uploading piece... Dec 26 13:29:32.284141471 (502h0m32.155224516s) [noise=true , address=77.90.31.11:4569 , piece_id=VHWTWGDHTLAJFEELYM5XNXOHXGFFDG2EBB2JXHAUUC57FLGQNQEA, seg_pos={0 0}, upload=440573]: Uploading piece... Dec 26 13:29:32.284487740 (502h0m32.155570775s) [noise=true , address=45.91.116.12:4235 , piece_id=2WQIOK5KRROZZOY63DDCT62IH5FQQMWKJZIX3YRRUJ6KCZ76THHQ, seg_pos={0 0}, upload=440573]: Uploading piece... Dec 26 13:29:32.284814412 (502h0m32.155897447s) [noise=true , address=77.90.31.16:4763 , piece_id=RPFTQSBCHPRKCO3WUXMZQP4LDWZB6HF6X3OL4Y5VHTXXUGLUC5HA, seg_pos={0 0}, upload=440573]: Uploading piece... Dec 26 13:29:33.755237461 (502h0m33.626320496s) [upload=440573]: Next returned nil segment Dec 26 13:29:33.755251126 (502h0m33.626334161s) [upload=440573]: Waiting for error group managing segments... Dec 26 13:29:34.167217093 (502h0m34.038300138s) [noise=true , address=145.79.227.40:4067 , piece_id=C5V5JGRNMF2AXAIYP5KBIEVTJFRK6R72A2X2EYQRYQH345EIYCMQ, seg_pos={0 0}, upload=440573]: Done uploading piece. err: <nil> Dec 26 13:29:34.173677816 (502h0m34.044760861s) [noise=true , address=145.79.227.52:4449 , piece_id=NZNYADQDANYMUQPZ26UFDXAB5CQF6ZZBQR3JCUR22FFA3WYH5FSA, seg_pos={0 0}, upload=440573]: Done uploading piece. err: <nil> Dec 26 13:29:34.174544977 (502h0m34.045628012s) [noise=true , address=145.79.227.48:4329 , piece_id=GLJAQM5YP3LCCSGHOUMQARORY6ZIGNIMJQPVFW56HHSHQKU3O64Q, seg_pos={0 0}, upload=440573]: Done uploading piece. err: <nil> Dec 26 13:29:34.193585975 (502h0m34.06466902s) [noise=true , address=145.79.227.29:4970 , piece_id=CMQTNSCP4DSIABKSNXYV4KSPRIVYBQ4YPU4ENLRVOXSUNIETQD6A, seg_pos={0 0}, upload=440573]: Done uploading piece. err: <nil> Dec 26 13:29:34.323466150 (502h0m34.194549185s) [noise=true , address=77.90.31.18:4810 , piece_id=R37QEJSHY47UUCI4IFQCVHEDUCE6AN6WU3VRNR67KXTL7EJ5VCJA, seg_pos={0 0}, upload=440573]: Done uploading piece. err: <nil> Dec 26 13:29:34.392077067 (502h0m34.263160102s) [noise=true , address=45.91.116.14:4195 , piece_id=GIU2CB4NSOCHLR4XBBB53GNZ4BYZUGDVZVPEPXA6DJQ5MH7EBIBA, seg_pos={0 0}, upload=440573]: Done uploading piece. err: <nil> Dec 26 13:29:34.439840176 (502h0m34.310923211s) [noise=true , address=77.90.31.19:4869 , piece_id=C2IYXUPGEBA7V7FOEIPOAQHVCZ5BICC7OS6DV6UBSGJXMMVDVRLA, seg_pos={0 0}, upload=440573]: Done uploading piece. err: <nil> Dec 26 13:29:34.486767434 (502h0m34.357850469s) [noise=true , address=77.90.31.15:4706 , piece_id=WPMHDF2AKEPXW5CN34DFH7T7PCQ4PRQUM5G6EFMQAS2HW3ITWPLA, seg_pos={0 0}, upload=440573]: Done uploading piece. err: <nil> Dec 26 13:29:34.507965730 (502h0m34.379048775s) [noise=true , address=77.90.31.11:4569 , piece_id=VHWTWGDHTLAJFEELYM5XNXOHXGFFDG2EBB2JXHAUUC57FLGQNQEA, seg_pos={0 0}, upload=440573]: Done uploading piece. err: <nil> Dec 26 13:29:34.528676206 (502h0m34.399759241s) [noise=true , address=45.91.116.7:4042 , piece_id=RQZSXDO6TETWGCIMF6CCTFETMOODBVIFIYXHZKDNSVW5BYBRZX4A, seg_pos={0 0}, upload=440573]: Done uploading piece. err: <nil> Dec 26 13:29:34.554686674 (502h0m34.42576971s) [noise=true , address=77.90.31.16:4763 , piece_id=RPFTQSBCHPRKCO3WUXMZQP4LDWZB6HF6X3OL4Y5VHTXXUGLUC5HA, seg_pos={0 0}, upload=440573]: Done uploading piece. err: <nil> Dec 26 13:29:34.817696482 (502h0m34.688779517s) [noise=true , address=77.90.31.12:4602 , piece_id=FLGZ4U6PWZNKH3FAS2YG3IAAA5XGMIBLBEYLCJ2PVGZDG6YBRD6Q, seg_pos={0 0}, upload=440573]: Done uploading piece. err: <nil> Dec 26 13:29:34.899012282 (502h0m34.770095328s) [noise=true , address=77.90.31.14:4671 , piece_id=TE3G3HW7P2JM3DUTBI44USPC3IWJMFZS3BY6JDFDZN2J4SMC4IHA, seg_pos={0 0}, upload=440573]: Done uploading piece. err: <nil> Dec 26 13:29:35.059110307 (502h0m34.930193352s) [noise=true , address=45.91.116.12:4235 , piece_id=2WQIOK5KRROZZOY63DDCT62IH5FQQMWKJZIX3YRRUJ6KCZ76THHQ, seg_pos={0 0}, upload=440573]: Done uploading piece. err: <nil> Dec 26 13:29:35.059135465 (502h0m34.93021851s) [seg_pos={0 0}, upload=440573]: Segment reached optimal threshold of 14 pieces. Dec 26 13:29:35.059365774 (502h0m34.930448809s) [noise=true , address=77.90.31.17:4776 , piece_id=4WY3HFUKERU6GJ6WWPEKUYI7DCLTYN5TXG6AHIJCBIMMYHYPCKRQ, seg_pos={0 0}, upload=440573]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Dec 26 13:29:35.059374851 (502h0m34.930457886s) [noise=true , address=77.90.31.17:4776 , piece_id=4WY3HFUKERU6GJ6WWPEKUYI7DCLTYN5TXG6AHIJCBIMMYHYPCKRQ, seg_pos={0 0}, upload=440573]: Long tail cancellation due to optimal threshold Dec 26 13:29:35.059671406 (502h0m34.930754451s) [noise=true , address=145.79.227.44:4205 , piece_id=5OSOHHKSVD3LKXPZ3HJFCWY6ZKWSTNCIKNFZTKVBME4CFZAHL3GA, seg_pos={0 0}, upload=440573]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UafxBVo9nuRVBcQmGE4gBP2pnXQ5YUrwHZdwPCeSBSBWCZKpQ): protocol: expected piece hash; context canceled; EOF Dec 26 13:29:35.059679611 (502h0m34.930762656s) [noise=true , address=145.79.227.44:4205 , piece_id=5OSOHHKSVD3LKXPZ3HJFCWY6ZKWSTNCIKNFZTKVBME4CFZAHL3GA, seg_pos={0 0}, upload=440573]: Long tail cancellation due to optimal threshold Dec 26 13:29:35.059826621 (502h0m34.930909656s) [noise=true , address=64.49.13.169:4706 , piece_id=ZHYAEU6IOT5FIRXRBNI2LMSWGVCORFL2MLUVEUTZDCPWYJ3NUETQ, seg_pos={0 0}, upload=440573]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:47026->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:47026->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47026->64.49.13.169:4706: use of closed network connection Dec 26 13:29:35.059834075 (502h0m34.93091713s) [noise=true , address=64.49.13.169:4706 , piece_id=ZHYAEU6IOT5FIRXRBNI2LMSWGVCORFL2MLUVEUTZDCPWYJ3NUETQ, seg_pos={0 0}, upload=440573]: Long tail cancellation due to optimal threshold Dec 26 13:29:35.060007676 (502h0m34.931090721s) [noise=true , address=145.79.227.30:4796 , piece_id=ITAXIGGPL3V7GFENXZHTI3P3UCVWKP6HP3RI2NHBHL7HCN2GR2JQ, seg_pos={0 0}, upload=440573]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 26 13:29:35.060016282 (502h0m34.931099317s) [noise=true , address=145.79.227.30:4796 , piece_id=ITAXIGGPL3V7GFENXZHTI3P3UCVWKP6HP3RI2NHBHL7HCN2GR2JQ, seg_pos={0 0}, upload=440573]: Long tail cancellation due to optimal threshold Dec 26 13:29:35.060036901 (502h0m34.931119947s) [seg_pos={0 0}, upload=440573]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 13:29:35.060109039 (502h0m34.931192084s) [upload=440573]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:29:35.060116573 (502h0m34.931199618s) [upload=440573]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:29:35.362502901 (502h0m35.233585936s) [upload=440573]: Done uploading segments... Dec 26 13:29:35.362548688 (502h0m35.233631733s) [upload=440573]: Upload finished. err: <nil> Duration: 1m0.003607598s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X.X..XX.....X..XXXXXX..X....X..X.X....X......XX.X.X......X.X] duration: 30.006069038s trace: https://jaeger.dev.storj.io/trace/1c0b85a1d6706345 FAIL (2025-12-26 13:30:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: R7DFCLKBBX27VDKRZXNFPM6WJKBPDHBZFTZMZZB7TJ2ISGQNCM6Q) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: N3HI2LUWCK642HS6WB46NBHMJMFAHW3LH6OGSOK56CQRIUV2W5MA) context deadline exceeded ======LOGS====== d Dec 26 13:29:59.128734722 (502h0m58.999817757s) [upload=440577]: Uploading segments... Dec 26 13:29:59.128766392 (502h0m58.999849427s) [upload=440577]: Got next segment. Inline: false Dec 26 13:29:59.128792242 (502h0m58.999875277s) [upload=440577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 13:29:59.128796550 (502h0m58.999879585s) [upload=440577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 13:29:59.373533585 (502h0m59.24461662s) [seg_pos={0 0}, upload=440577]: Begin upload segment... Dec 26 13:29:59.373579592 (502h0m59.244662627s) [seg_pos={0 0}, upload=440577]: Done begin upload segment. Dec 26 13:29:59.373604510 (502h0m59.244687545s) [noise=true , address=145.79.227.35:4483 , piece_id=5TFECXO62MBXQDCTL7XH5QX7AMRKDMSWNEWQXQ6LDCA4HJRLP4UQ, seg_pos={0 0}, upload=440577]: Uploading piece... Dec 26 13:29:59.373635088 (502h0m59.244718123s) [noise=true , address=45.91.116.11:4118 , piece_id=WTOL5RDDLUTWVPCZNHYESH3FOVKF2ROP62H4B7LAHKWGBGOAMMUA, seg_pos={0 0}, upload=440577]: Uploading piece... Dec 26 13:29:59.374131082 (502h0m59.245214117s) [noise=true , address=145.79.227.17:4970 , piece_id=WFMPDUQA6VB67Q4KFT7G6DOENDKF5AGNTNUQ35AKBBZ4HKFY5RRQ, seg_pos={0 0}, upload=440577]: Uploading piece... Dec 26 13:29:59.374410124 (502h0m59.245493159s) [noise=true , address=145.79.227.37:4567 , piece_id=IYZD7BEHY4GHAX54MTHL7ETT4SRREIAXKYYVXUYNY6VZLYPF27VQ, seg_pos={0 0}, upload=440577]: Uploading piece... Dec 26 13:29:59.374710787 (502h0m59.245793822s) [noise=true , address=145.79.227.16:4910 , piece_id=6QBNN3C36DS2W7JGDGW235ZDEQQFNPTLR4WQDSXB4JHSJ5I67PGQ, seg_pos={0 0}, upload=440577]: Uploading piece... Dec 26 13:29:59.374987624 (502h0m59.246070659s) [noise=true , address=45.91.116.10:4082 , piece_id=7GQEWAVI6XPBC2CBNCSUXG76HWM5CY53H4JHNBGBKDEL4C7MK7OA, seg_pos={0 0}, upload=440577]: Uploading piece... Dec 26 13:29:59.375272767 (502h0m59.246355802s) [noise=true , address=45.91.116.5:4300 , piece_id=LIQNDNL2VMBKUKN5LOQEJMAEUDYPPWWPAQIFV74QTIHACLRGVRHQ, seg_pos={0 0}, upload=440577]: Uploading piece... Dec 26 13:29:59.375520398 (502h0m59.246603433s) [noise=true , address=77.90.31.14:4663 , piece_id=N3HI2LUWCK642HS6WB46NBHMJMFAHW3LH6OGSOK56CQRIUV2W5MA, seg_pos={0 0}, upload=440577]: Uploading piece... Dec 26 13:29:59.375811041 (502h0m59.246894086s) [noise=true , address=145.79.227.4:4186 , piece_id=7PKODQ5C3RCWU62OAMFVHNVTDNWZM6HLPSXAWPERGRYNIWHTDXUQ, seg_pos={0 0}, upload=440577]: Uploading piece... Dec 26 13:29:59.376094110 (502h0m59.247177145s) [noise=true , address=145.79.227.48:4302 , piece_id=YCYMVKWFYCQGAML7BXUUPTPSNB7E4LWDOQX46ABEWZIFHWWPLIUA, seg_pos={0 0}, upload=440577]: Uploading piece... Dec 26 13:29:59.377954223 (502h0m59.249037268s) [noise=true , address=145.79.227.29:4941 , piece_id=6IFO337QZWSREFCZAH35ULLVILC6SG4V4VXKRZDVYHXUJUDGWY6A, seg_pos={0 0}, upload=440577]: Uploading piece... Dec 26 13:29:59.378312414 (502h0m59.249395449s) [noise=true , address=77.90.31.17:4773 , piece_id=KQI4XXGDAXC6IKX25L5MUXZCH4TV5YCXTCWKLCBLJVOJ2C72RE4A, seg_pos={0 0}, upload=440577]: Uploading piece... Dec 26 13:29:59.378659445 (502h0m59.24974248s) [noise=true , address=77.90.31.16:4735 , piece_id=R7DFCLKBBX27VDKRZXNFPM6WJKBPDHBZFTZMZZB7TJ2ISGQNCM6Q, seg_pos={0 0}, upload=440577]: Uploading piece... Dec 26 13:29:59.379690689 (502h0m59.250773724s) [noise=true , address=45.91.116.4:4939 , piece_id=BPMADH3DHW5GXDNE5KTAU2RJZTE3D3DVTJJWN73M3QKB73FMRWDA, seg_pos={0 0}, upload=440577]: Uploading piece... Dec 26 13:29:59.380824437 (502h0m59.251907473s) [noise=true , address=145.79.227.22:4511 , piece_id=S4TQXZ2PQAECOEUCRA735TSQCHMEJENKSCX3BLXN7RLFD6P5E6HQ, seg_pos={0 0}, upload=440577]: Uploading piece... Dec 26 13:29:59.382145864 (502h0m59.253228899s) [noise=true , address=45.91.116.9:4261 , piece_id=JUJEWFFPAVRWKUSA47NDQBIAP7DO6DWVCMXPIVZWBIPTWS5W4DXQ, seg_pos={0 0}, upload=440577]: Uploading piece... Dec 26 13:30:01.100561450 (502h1m0.971644485s) [upload=440577]: Next returned nil segment Dec 26 13:30:01.100583282 (502h1m0.971666317s) [upload=440577]: Waiting for error group managing segments... Dec 26 13:30:01.377491026 (502h1m1.248574071s) [noise=true , address=145.79.227.48:4302 , piece_id=YCYMVKWFYCQGAML7BXUUPTPSNB7E4LWDOQX46ABEWZIFHWWPLIUA, seg_pos={0 0}, upload=440577]: Done uploading piece. err: <nil> Dec 26 13:30:01.388085129 (502h1m1.259168174s) [noise=true , address=145.79.227.29:4941 , piece_id=6IFO337QZWSREFCZAH35ULLVILC6SG4V4VXKRZDVYHXUJUDGWY6A, seg_pos={0 0}, upload=440577]: Done uploading piece. err: <nil> Dec 26 13:30:01.407082083 (502h1m1.278165128s) [noise=true , address=145.79.227.37:4567 , piece_id=IYZD7BEHY4GHAX54MTHL7ETT4SRREIAXKYYVXUYNY6VZLYPF27VQ, seg_pos={0 0}, upload=440577]: Done uploading piece. err: <nil> Dec 26 13:30:01.408405583 (502h1m1.279488638s) [noise=true , address=145.79.227.17:4970 , piece_id=WFMPDUQA6VB67Q4KFT7G6DOENDKF5AGNTNUQ35AKBBZ4HKFY5RRQ, seg_pos={0 0}, upload=440577]: Done uploading piece. err: <nil> Dec 26 13:30:01.411481089 (502h1m1.282564134s) [noise=true , address=145.79.227.16:4910 , piece_id=6QBNN3C36DS2W7JGDGW235ZDEQQFNPTLR4WQDSXB4JHSJ5I67PGQ, seg_pos={0 0}, upload=440577]: Done uploading piece. err: <nil> Dec 26 13:30:01.412343121 (502h1m1.283426166s) [noise=true , address=145.79.227.22:4511 , piece_id=S4TQXZ2PQAECOEUCRA735TSQCHMEJENKSCX3BLXN7RLFD6P5E6HQ, seg_pos={0 0}, upload=440577]: Done uploading piece. err: <nil> Dec 26 13:30:01.414364319 (502h1m1.285447364s) [noise=true , address=145.79.227.35:4483 , piece_id=5TFECXO62MBXQDCTL7XH5QX7AMRKDMSWNEWQXQ6LDCA4HJRLP4UQ, seg_pos={0 0}, upload=440577]: Done uploading piece. err: <nil> Dec 26 13:30:01.421545965 (502h1m1.292629s) [noise=true , address=145.79.227.4:4186 , piece_id=7PKODQ5C3RCWU62OAMFVHNVTDNWZM6HLPSXAWPERGRYNIWHTDXUQ, seg_pos={0 0}, upload=440577]: Done uploading piece. err: <nil> Dec 26 13:30:01.599616284 (502h1m1.470699329s) [noise=true , address=77.90.31.14:4663 , piece_id=N3HI2LUWCK642HS6WB46NBHMJMFAHW3LH6OGSOK56CQRIUV2W5MA, seg_pos={0 0}, upload=440577]: Done uploading piece. err: <nil> Dec 26 13:30:01.612627741 (502h1m1.483710786s) [noise=true , address=77.90.31.16:4735 , piece_id=R7DFCLKBBX27VDKRZXNFPM6WJKBPDHBZFTZMZZB7TJ2ISGQNCM6Q, seg_pos={0 0}, upload=440577]: Done uploading piece. err: <nil> Dec 26 13:30:01.706133695 (502h1m1.57721675s) [noise=true , address=45.91.116.4:4939 , piece_id=BPMADH3DHW5GXDNE5KTAU2RJZTE3D3DVTJJWN73M3QKB73FMRWDA, seg_pos={0 0}, upload=440577]: Done uploading piece. err: <nil> Dec 26 13:30:01.768761910 (502h1m1.639844965s) [noise=true , address=77.90.31.17:4773 , piece_id=KQI4XXGDAXC6IKX25L5MUXZCH4TV5YCXTCWKLCBLJVOJ2C72RE4A, seg_pos={0 0}, upload=440577]: Done uploading piece. err: <nil> Dec 26 13:30:01.819194072 (502h1m1.690277117s) [noise=true , address=45.91.116.11:4118 , piece_id=WTOL5RDDLUTWVPCZNHYESH3FOVKF2ROP62H4B7LAHKWGBGOAMMUA, seg_pos={0 0}, upload=440577]: Done uploading piece. err: <nil> Dec 26 13:30:01.819232816 (502h1m1.690315861s) [seg_pos={0 0}, upload=440577]: Segment reached optimal threshold of 13 pieces. Dec 26 13:30:01.819483684 (502h1m1.690566739s) [noise=true , address=45.91.116.9:4261 , piece_id=JUJEWFFPAVRWKUSA47NDQBIAP7DO6DWVCMXPIVZWBIPTWS5W4DXQ, seg_pos={0 0}, upload=440577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 26 13:30:01.819494404 (502h1m1.690577449s) [noise=true , address=45.91.116.9:4261 , piece_id=JUJEWFFPAVRWKUSA47NDQBIAP7DO6DWVCMXPIVZWBIPTWS5W4DXQ, seg_pos={0 0}, upload=440577]: Long tail cancellation due to optimal threshold Dec 26 13:30:01.819949741 (502h1m1.691032786s) [noise=true , address=45.91.116.10:4082 , piece_id=7GQEWAVI6XPBC2CBNCSUXG76HWM5CY53H4JHNBGBKDEL4C7MK7OA, seg_pos={0 0}, upload=440577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:37612->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:37612->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37612->45.91.116.10:4082: use of closed network connection Dec 26 13:30:01.819962285 (502h1m1.69104533s) [noise=true , address=45.91.116.10:4082 , piece_id=7GQEWAVI6XPBC2CBNCSUXG76HWM5CY53H4JHNBGBKDEL4C7MK7OA, seg_pos={0 0}, upload=440577]: Long tail cancellation due to optimal threshold Dec 26 13:30:01.820110167 (502h1m1.691193212s) [noise=true , address=45.91.116.5:4300 , piece_id=LIQNDNL2VMBKUKN5LOQEJMAEUDYPPWWPAQIFV74QTIHACLRGVRHQ, seg_pos={0 0}, upload=440577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:37800->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:37800->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37800->45.91.116.5:4300: use of closed network connection Dec 26 13:30:01.820120165 (502h1m1.691203211s) [noise=true , address=45.91.116.5:4300 , piece_id=LIQNDNL2VMBKUKN5LOQEJMAEUDYPPWWPAQIFV74QTIHACLRGVRHQ, seg_pos={0 0}, upload=440577]: Long tail cancellation due to optimal threshold Dec 26 13:30:01.820141887 (502h1m1.691224932s) [seg_pos={0 0}, upload=440577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 13:30:01.820237539 (502h1m1.691320584s) [upload=440577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:30:01.820245745 (502h1m1.69132879s) [upload=440577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:30:02.111713364 (502h1m1.982796409s) [upload=440577]: Done uploading segments... Dec 26 13:30:02.111742239 (502h1m1.982825284s) [upload=440577]: Upload finished. err: <nil> Duration: 30.006085299s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [XXXX.XXXX.X..X..X....XXXXXX...XX.....X...XX.X.XXX.XXXXX..X.X] duration: 5.001811367s trace: https://jaeger.dev.storj.io/trace/7db04fcc87596bb8 FAIL (2025-12-26 13:30:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001838829s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [XXXXX.XXXX.X..X..X....XXXXXX...XX.....X...XX.X.XXX.XXXXX..X.] duration: 4.411306076s trace: https://jaeger.dev.storj.io/trace/17542e8e9ef9d6ac FAIL (2025-12-26 13:29:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.411323989s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X.X..XX.....X..XXXXXX..X....X..X.X....X......XX.X.X......X.] duration: 6.636102936s trace: https://jaeger.dev.storj.io/trace/2c4049578a7dd358 FAIL (2025-12-26 13:29:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 13:28:26.478576262 (501h59m26.349659297s) [upload=440556]: Starting upload Dec 26 13:28:26.478673758 (501h59m26.349756793s) [upload=440556]: Done starting upload Dec 26 13:28:26.479551479 (501h59m26.350634524s) [upload=440556]: Uploading segments... Dec 26 13:28:26.479623476 (501h59m26.350706521s) [upload=440556]: Got next segment. Inline: false Dec 26 13:28:26.479661429 (501h59m26.350744464s) [upload=440556]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 13:28:26.479666840 (501h59m26.350749885s) [upload=440556]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 13:28:26.732817332 (501h59m26.603900377s) [seg_pos={0 0}, upload=440556]: Begin upload segment... Dec 26 13:28:26.732882195 (501h59m26.60396523s) [seg_pos={0 0}, upload=440556]: Done begin upload segment. Dec 26 13:28:26.732942791 (501h59m26.604025836s) [noise=true , address=145.79.227.15:4873 , piece_id=TQTIAWBDYZVVPB7SX62D5JDXDGQ6AHVTVRI6IKEYNNJSVKS4AAZQ, seg_pos={0 0}, upload=440556]: Uploading piece... Dec 26 13:28:26.732957549 (501h59m26.604040594s) [noise=true , address=145.79.227.47:4267 , piece_id=LMRCHJSR3TJ54SCKIX6CILAYSB4RC2ZWEDY6CJBRDQWHQZFI7DTA, seg_pos={0 0}, upload=440556]: Uploading piece... Dec 26 13:28:26.733487018 (501h59m26.604570053s) [noise=true , address=145.79.227.17:4971 , piece_id=UK5AWRRDQH7R6HTVPYUZY5VK4EV3FE6FRNHPZHSV3T6PFEPYQXVA, seg_pos={0 0}, upload=440556]: Uploading piece... Dec 26 13:28:26.733988302 (501h59m26.605071347s) [noise=true , address=145.79.227.45:4230 , piece_id=72MSL2QUWLBWQX4NGFBKFFAGHMMBXB2VFPD5WJIEOFLURMXO272Q, seg_pos={0 0}, upload=440556]: Uploading piece... Dec 26 13:28:26.733993812 (501h59m26.605076847s) [noise=true , address=45.91.116.11:4118 , piece_id=66OSTMYR26WFAIFH2PH5CTFZPNZOKZWLWGLHNFRDRXJATSSFJYGA, seg_pos={0 0}, upload=440556]: Uploading piece... Dec 26 13:28:26.734278724 (501h59m26.605361759s) [noise=true , address=77.90.31.15:4700 , piece_id=OKO5DCA5PMLGN5AITPRT56EOM4ARFFZDG4XEX4FG7ZZBH5GKQUOQ, seg_pos={0 0}, upload=440556]: Uploading piece... Dec 26 13:28:26.734713523 (501h59m26.605796558s) [noise=true , address=77.90.31.17:4773 , piece_id=I6XB5GB7WFCW53QDFUBMHG4DPGMPSXFMRIEL5LOS2WDY3YPJ6M7Q, seg_pos={0 0}, upload=440556]: Uploading piece... Dec 26 13:28:26.735059502 (501h59m26.606142537s) [noise=true , address=145.79.227.26:4654 , piece_id=6HKUX4UHR7VH7MPECJ6CCSRW5JKR4YUTVEQ6IRYJ4EVRAHRD5YRA, seg_pos={0 0}, upload=440556]: Uploading piece... Dec 26 13:28:26.736352133 (501h59m26.607435168s) [noise=true , address=145.79.227.42:4121 , piece_id=X7ZRASOCBF4P7OKT2VF3PFBABMCPYCTYH5IMTHEEJ2QXAYQ7KA7Q, seg_pos={0 0}, upload=440556]: Uploading piece... Dec 26 13:28:26.737591724 (501h59m26.608674769s) [noise=true , address=145.79.227.35:4482 , piece_id=ZBKK6FM5D4FS6M2RP5MTTXHZ4WPB5TRQA5ITJYICXNRGFTFIPIXA, seg_pos={0 0}, upload=440556]: Uploading piece... Dec 26 13:28:26.738943206 (501h59m26.610026251s) [noise=true , address=77.90.31.14:4661 , piece_id=WVB67BKU3ZW2FCISXMHRBHRT25GP74QL2T4VSJW3T3HPGNOIOTFQ, seg_pos={0 0}, upload=440556]: Uploading piece... Dec 26 13:28:26.739680591 (501h59m26.610763636s) [noise=true , address=45.91.116.8:4046 , piece_id=HIMOMLCGOCYWJHLYHVXWFGVGC45X3Y4KSMPERRT4PXJNQVZCWS7A, seg_pos={0 0}, upload=440556]: Uploading piece... Dec 26 13:28:26.740045676 (501h59m26.611128711s) [noise=true , address=145.79.227.38:4977 , piece_id=PR7AAZPMNLGNGNNOESE5MKZOEZLKISSZHZEEXELY64EU6ERFOJTA, seg_pos={0 0}, upload=440556]: Uploading piece... Dec 26 13:28:26.740463872 (501h59m26.611546907s) [noise=true , address=145.79.227.3:4149 , piece_id=5CZKG5SWHLMEGK5AGRXWFD5ZQTAFW4XOSMH5YUXQHFW2APCTRIXA, seg_pos={0 0}, upload=440556]: Uploading piece... Dec 26 13:28:26.741487120 (501h59m26.612570165s) [noise=true , address=45.91.116.6:4974 , piece_id=KPFPHATPLSVMQNE6S2RZ7PLXVIJQL3D64H2MGTEGZD3BSIGADZYA, seg_pos={0 0}, upload=440556]: Uploading piece... Dec 26 13:28:26.741973356 (501h59m26.613056391s) [noise=true , address=145.79.227.37:4566 , piece_id=KSCLCC43JWOY7SEBPVKCEFZ6R4ILGL37AYBY6XL5XFQLR47Y22KQ, seg_pos={0 0}, upload=440556]: Uploading piece... Dec 26 13:28:28.447582239 (501h59m28.318665284s) [upload=440556]: Next returned nil segment Dec 26 13:28:28.447598220 (501h59m28.318681255s) [upload=440556]: Waiting for error group managing segments... Dec 26 13:28:28.736676457 (501h59m28.607759492s) [noise=true , address=145.79.227.38:4977 , piece_id=PR7AAZPMNLGNGNNOESE5MKZOEZLKISSZHZEEXELY64EU6ERFOJTA, seg_pos={0 0}, upload=440556]: Done uploading piece. err: <nil> Dec 26 13:28:28.745907294 (501h59m28.616990349s) [noise=true , address=145.79.227.17:4971 , piece_id=UK5AWRRDQH7R6HTVPYUZY5VK4EV3FE6FRNHPZHSV3T6PFEPYQXVA, seg_pos={0 0}, upload=440556]: Done uploading piece. err: <nil> Dec 26 13:28:28.750438662 (501h59m28.621521707s) [noise=true , address=145.79.227.42:4121 , piece_id=X7ZRASOCBF4P7OKT2VF3PFBABMCPYCTYH5IMTHEEJ2QXAYQ7KA7Q, seg_pos={0 0}, upload=440556]: Done uploading piece. err: <nil> Dec 26 13:28:28.754524613 (501h59m28.625607648s) [noise=true , address=145.79.227.26:4654 , piece_id=6HKUX4UHR7VH7MPECJ6CCSRW5JKR4YUTVEQ6IRYJ4EVRAHRD5YRA, seg_pos={0 0}, upload=440556]: Done uploading piece. err: <nil> Dec 26 13:28:28.754845073 (501h59m28.625928108s) [noise=true , address=145.79.227.45:4230 , piece_id=72MSL2QUWLBWQX4NGFBKFFAGHMMBXB2VFPD5WJIEOFLURMXO272Q, seg_pos={0 0}, upload=440556]: Done uploading piece. err: <nil> Dec 26 13:28:28.763236261 (501h59m28.634319296s) [noise=true , address=145.79.227.3:4149 , piece_id=5CZKG5SWHLMEGK5AGRXWFD5ZQTAFW4XOSMH5YUXQHFW2APCTRIXA, seg_pos={0 0}, upload=440556]: Done uploading piece. err: <nil> Dec 26 13:28:28.768839691 (501h59m28.639922726s) [noise=true , address=145.79.227.15:4873 , piece_id=TQTIAWBDYZVVPB7SX62D5JDXDGQ6AHVTVRI6IKEYNNJSVKS4AAZQ, seg_pos={0 0}, upload=440556]: Done uploading piece. err: <nil> Dec 26 13:28:28.770801837 (501h59m28.641884872s) [noise=true , address=145.79.227.35:4482 , piece_id=ZBKK6FM5D4FS6M2RP5MTTXHZ4WPB5TRQA5ITJYICXNRGFTFIPIXA, seg_pos={0 0}, upload=440556]: Done uploading piece. err: <nil> Dec 26 13:28:28.773115814 (501h59m28.644198859s) [noise=true , address=145.79.227.37:4566 , piece_id=KSCLCC43JWOY7SEBPVKCEFZ6R4ILGL37AYBY6XL5XFQLR47Y22KQ, seg_pos={0 0}, upload=440556]: Done uploading piece. err: <nil> Dec 26 13:28:28.777789874 (501h59m28.648872919s) [noise=true , address=145.79.227.47:4267 , piece_id=LMRCHJSR3TJ54SCKIX6CILAYSB4RC2ZWEDY6CJBRDQWHQZFI7DTA, seg_pos={0 0}, upload=440556]: Done uploading piece. err: <nil> Dec 26 13:28:28.976158824 (501h59m28.847241859s) [noise=true , address=77.90.31.14:4661 , piece_id=WVB67BKU3ZW2FCISXMHRBHRT25GP74QL2T4VSJW3T3HPGNOIOTFQ, seg_pos={0 0}, upload=440556]: Done uploading piece. err: <nil> Dec 26 13:28:28.989780291 (501h59m28.860863326s) [noise=true , address=45.91.116.11:4118 , piece_id=66OSTMYR26WFAIFH2PH5CTFZPNZOKZWLWGLHNFRDRXJATSSFJYGA, seg_pos={0 0}, upload=440556]: Done uploading piece. err: <nil> Dec 26 13:28:29.044241055 (501h59m28.91532409s) [noise=true , address=77.90.31.15:4700 , piece_id=OKO5DCA5PMLGN5AITPRT56EOM4ARFFZDG4XEX4FG7ZZBH5GKQUOQ, seg_pos={0 0}, upload=440556]: Done uploading piece. err: <nil> Dec 26 13:28:29.044261454 (501h59m28.915344499s) [seg_pos={0 0}, upload=440556]: Segment reached optimal threshold of 13 pieces. Dec 26 13:28:29.044559682 (501h59m28.915642727s) [noise=true , address=77.90.31.17:4773 , piece_id=I6XB5GB7WFCW53QDFUBMHG4DPGMPSXFMRIEL5LOS2WDY3YPJ6M7Q, seg_pos={0 0}, upload=440556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 26 13:28:29.044579469 (501h59m28.915662514s) [noise=true , address=77.90.31.17:4773 , piece_id=I6XB5GB7WFCW53QDFUBMHG4DPGMPSXFMRIEL5LOS2WDY3YPJ6M7Q, seg_pos={0 0}, upload=440556]: Long tail cancellation due to optimal threshold Dec 26 13:28:29.044590530 (501h59m28.915673595s) [noise=true , address=45.91.116.8:4046 , piece_id=HIMOMLCGOCYWJHLYHVXWFGVGC45X3Y4KSMPERRT4PXJNQVZCWS7A, seg_pos={0 0}, upload=440556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 26 13:28:29.044601261 (501h59m28.915684296s) [noise=true , address=45.91.116.8:4046 , piece_id=HIMOMLCGOCYWJHLYHVXWFGVGC45X3Y4KSMPERRT4PXJNQVZCWS7A, seg_pos={0 0}, upload=440556]: Long tail cancellation due to optimal threshold Dec 26 13:28:29.044850997 (501h59m28.915934032s) [noise=true , address=45.91.116.6:4974 , piece_id=KPFPHATPLSVMQNE6S2RZ7PLXVIJQL3D64H2MGTEGZD3BSIGADZYA, seg_pos={0 0}, upload=440556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 26 13:28:29.044859032 (501h59m28.915942067s) [noise=true , address=45.91.116.6:4974 , piece_id=KPFPHATPLSVMQNE6S2RZ7PLXVIJQL3D64H2MGTEGZD3BSIGADZYA, seg_pos={0 0}, upload=440556]: Long tail cancellation due to optimal threshold Dec 26 13:28:29.044881825 (501h59m28.91596486s) [seg_pos={0 0}, upload=440556]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 13:28:29.044963060 (501h59m28.916046095s) [upload=440556]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:28:29.044968971 (501h59m28.916052016s) [upload=440556]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:28:29.334823425 (501h59m29.20590647s) [upload=440556]: Done uploading segments... Dec 26 13:28:29.334847151 (501h59m29.205930186s) [upload=440556]: Upload finished. err: <nil> Duration: 6.636128876s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 15 out of 60 is failed (threshold=3) [......X.XXX...X..X...................XX.X........X.XXX.X.X..] duration: 47.477208594s trace: https://jaeger.dev.storj.io/trace/544d4bfd645cbdde FAIL (2025-12-26 13:29:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, Piece ID: CY6ASE2FHX3MVI7J65FAWX2GHNBD3PV426P4C7ZGMCMEDLPPDGOA) context deadline exceeded; piecestore: (Node ID: 1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, Piece ID: ZVGVUIZE35ODNIVTVKDLGYWTNKI6H2ZWXWKYV6I2FSWTAEVFGNXA) context deadline exceeded; piecestore: (Node ID: 1ZgtJJK3pt2NqSfgFnXjZiYhDafhRdh6WkEf12AFTTuiku3Uhc, Piece ID: 7X63XPGXFUDIK34XF4DMJPYPLQAVWIDITBJXLCMX4QZCAA3DRHOA) context deadline exceeded ======LOGS====== QOKA, seg_pos={0 0}, upload=440528]: Uploading piece... Dec 26 13:26:42.753351590 (501h57m42.624434625s) [noise=true , address=77.90.31.15:4703 , piece_id=RM36WNUNP2BS5QHE4TGGRVKB4JVYTGRWDZMYJEMAJHUT6H6EGYGA, seg_pos={0 0}, upload=440528]: Uploading piece... Dec 26 13:26:42.754332638 (501h57m42.625415673s) [noise=true , address=77.90.31.13:4631 , piece_id=CY6ASE2FHX3MVI7J65FAWX2GHNBD3PV426P4C7ZGMCMEDLPPDGOA, seg_pos={0 0}, upload=440528]: Uploading piece... Dec 26 13:26:42.755061256 (501h57m42.626144291s) [noise=true , address=45.91.116.8:4062 , piece_id=7X63XPGXFUDIK34XF4DMJPYPLQAVWIDITBJXLCMX4QZCAA3DRHOA, seg_pos={0 0}, upload=440528]: Uploading piece... Dec 26 13:26:42.756415695 (501h57m42.62749873s) [noise=true , address=145.79.227.44:4215 , piece_id=3D7KI7MIPA3VM7U5QHHRPDMEB6SZYVXGTSC3KVJLWNNJQZC7YBJA, seg_pos={0 0}, upload=440528]: Uploading piece... Dec 26 13:26:42.758023246 (501h57m42.629106291s) [noise=true , address=64.49.13.155:4922 , piece_id=OK2XBATJRCLLDM3Y3HTRUUI3VCVSQKMGA6UT3XTQNVXAHZLKRTLA, seg_pos={0 0}, upload=440528]: Uploading piece... Dec 26 13:26:42.759193595 (501h57m42.63027663s) [noise=true , address=77.90.31.10:4546 , piece_id=ZVGVUIZE35ODNIVTVKDLGYWTNKI6H2ZWXWKYV6I2FSWTAEVFGNXA, seg_pos={0 0}, upload=440528]: Uploading piece... Dec 26 13:26:42.760165165 (501h57m42.6312482s) [noise=true , address=77.90.31.17:4794 , piece_id=ZCCSOTOPZ4TB3V6R7T7C4ZN5ICF4H7CLPJGSQNJQ6LR47LT2ZLWA, seg_pos={0 0}, upload=440528]: Uploading piece... Dec 26 13:26:42.761242336 (501h57m42.632325381s) [noise=true , address=145.79.227.30:4784 , piece_id=5JTNMIH7VY2PHJTAHAW4OBXCAHLOLIYUERA4W77INMBBHPRQXD5Q, seg_pos={0 0}, upload=440528]: Uploading piece... Dec 26 13:26:42.761742378 (501h57m42.632825413s) [noise=true , address=145.79.227.40:4082 , piece_id=7A76MVACCKAZKNBVKOAN4THEEE2EGHR3BORKKHURPTYGUNU7VNRA, seg_pos={0 0}, upload=440528]: Uploading piece... Dec 26 13:26:42.762530559 (501h57m42.633613604s) [noise=true , address=77.90.31.14:4673 , piece_id=5WRLL4ABQX6X7OCOUJWU6LXNUBRBQZ6UOHPLA6ZEG5SRCXDLWLBQ, seg_pos={0 0}, upload=440528]: Uploading piece... Dec 26 13:26:42.763791189 (501h57m42.634874234s) [noise=true , address=145.79.227.49:4354 , piece_id=5P6JE3TVBJP5PGN6E5S3M4EPX4LGYFSJDDYC2Z2XIJDIVBUH645Q, seg_pos={0 0}, upload=440528]: Uploading piece... Dec 26 13:26:42.764363079 (501h57m42.635446114s) [noise=true , address=77.90.31.21:4907 , piece_id=4PSXBCAQFZIL6RK6X53KDYME6REMVIVBWCBYIUF75NEZN65XTD7Q, seg_pos={0 0}, upload=440528]: Uploading piece... Dec 26 13:26:42.764789631 (501h57m42.635872676s) [noise=true , address=193.149.164.143:4702 , piece_id=IB4KIMIAIHC632QAE36MLWOMPZNMLBYNLHSFELLYXKWGBLDWG2HQ, seg_pos={0 0}, upload=440528]: Uploading piece... Dec 26 13:26:42.765162862 (501h57m42.636245897s) [noise=true , address=145.79.227.29:4976 , piece_id=K7RGYM2EHVRE5HP654YLRHCHG2MHK4LSLN3HPGPBQ4I2PZIFWIKA, seg_pos={0 0}, upload=440528]: Uploading piece... Dec 26 13:26:42.765718189 (501h57m42.636801234s) [noise=true , address=64.49.13.138:4491 , piece_id=BCFZONIQTTWSXHKCJNILG5TVNTKY6QFLONU57Q7B6SSU5F2CCBUQ, seg_pos={0 0}, upload=440528]: Uploading piece... Dec 26 13:26:44.127815084 (501h57m43.998898119s) [upload=440528]: Next returned nil segment Dec 26 13:26:44.127826767 (501h57m43.998909802s) [upload=440528]: Waiting for error group managing segments... Dec 26 13:26:44.488486137 (501h57m44.359569182s) [noise=true , address=64.49.13.138:4491 , piece_id=BCFZONIQTTWSXHKCJNILG5TVNTKY6QFLONU57Q7B6SSU5F2CCBUQ, seg_pos={0 0}, upload=440528]: Done uploading piece. err: <nil> Dec 26 13:26:44.499128032 (501h57m44.370211067s) [noise=true , address=64.49.13.155:4922 , piece_id=OK2XBATJRCLLDM3Y3HTRUUI3VCVSQKMGA6UT3XTQNVXAHZLKRTLA, seg_pos={0 0}, upload=440528]: Done uploading piece. err: <nil> Dec 26 13:26:44.659480261 (501h57m44.530563296s) [noise=true , address=145.79.227.49:4354 , piece_id=5P6JE3TVBJP5PGN6E5S3M4EPX4LGYFSJDDYC2Z2XIJDIVBUH645Q, seg_pos={0 0}, upload=440528]: Done uploading piece. err: <nil> Dec 26 13:26:44.663261149 (501h57m44.534344184s) [noise=true , address=145.79.227.29:4976 , piece_id=K7RGYM2EHVRE5HP654YLRHCHG2MHK4LSLN3HPGPBQ4I2PZIFWIKA, seg_pos={0 0}, upload=440528]: Done uploading piece. err: <nil> Dec 26 13:26:44.669270713 (501h57m44.540353748s) [noise=true , address=145.79.227.43:4172 , piece_id=PUWRPRUKWCOZQBRC75BXSS6YU2LF3H5F6JTUPZ35VHOHIICCY4ZA, seg_pos={0 0}, upload=440528]: Done uploading piece. err: <nil> Dec 26 13:26:44.794603860 (501h57m44.665686905s) [noise=true , address=77.90.31.16:4761 , piece_id=QWGK5WGHM4LKV4Z4XIN2AWTQHD37U6ZGTLZP7M26J4DPBAIHQOKA, seg_pos={0 0}, upload=440528]: Done uploading piece. err: <nil> Dec 26 13:26:44.825995796 (501h57m44.697078831s) [noise=true , address=77.90.31.21:4907 , piece_id=4PSXBCAQFZIL6RK6X53KDYME6REMVIVBWCBYIUF75NEZN65XTD7Q, seg_pos={0 0}, upload=440528]: Done uploading piece. err: <nil> Dec 26 13:26:44.944381634 (501h57m44.815464669s) [noise=true , address=77.90.31.10:4546 , piece_id=ZVGVUIZE35ODNIVTVKDLGYWTNKI6H2ZWXWKYV6I2FSWTAEVFGNXA, seg_pos={0 0}, upload=440528]: Done uploading piece. err: <nil> Dec 26 13:26:45.061028420 (501h57m44.932111465s) [noise=true , address=77.90.31.15:4703 , piece_id=RM36WNUNP2BS5QHE4TGGRVKB4JVYTGRWDZMYJEMAJHUT6H6EGYGA, seg_pos={0 0}, upload=440528]: Done uploading piece. err: <nil> Dec 26 13:26:45.143197956 (501h57m45.014280991s) [noise=true , address=77.90.31.13:4631 , piece_id=CY6ASE2FHX3MVI7J65FAWX2GHNBD3PV426P4C7ZGMCMEDLPPDGOA, seg_pos={0 0}, upload=440528]: Done uploading piece. err: <nil> Dec 26 13:26:45.209300737 (501h57m45.080383772s) [noise=true , address=45.91.116.8:4062 , piece_id=7X63XPGXFUDIK34XF4DMJPYPLQAVWIDITBJXLCMX4QZCAA3DRHOA, seg_pos={0 0}, upload=440528]: Done uploading piece. err: <nil> Dec 26 13:26:45.609953281 (501h57m45.481036316s) [noise=true , address=145.79.227.44:4215 , piece_id=3D7KI7MIPA3VM7U5QHHRPDMEB6SZYVXGTSC3KVJLWNNJQZC7YBJA, seg_pos={0 0}, upload=440528]: Done uploading piece. err: <nil> Dec 26 13:26:45.648730770 (501h57m45.519813805s) [noise=true , address=145.79.227.40:4082 , piece_id=7A76MVACCKAZKNBVKOAN4THEEE2EGHR3BORKKHURPTYGUNU7VNRA, seg_pos={0 0}, upload=440528]: Done uploading piece. err: <nil> Dec 26 13:26:45.660324807 (501h57m45.531407842s) [noise=true , address=193.149.164.143:4702 , piece_id=IB4KIMIAIHC632QAE36MLWOMPZNMLBYNLHSFELLYXKWGBLDWG2HQ, seg_pos={0 0}, upload=440528]: Done uploading piece. err: <nil> Dec 26 13:26:45.660349645 (501h57m45.53143269s) [seg_pos={0 0}, upload=440528]: Segment reached optimal threshold of 14 pieces. Dec 26 13:26:45.660606213 (501h57m45.531689248s) [noise=true , address=145.79.227.30:4784 , piece_id=5JTNMIH7VY2PHJTAHAW4OBXCAHLOLIYUERA4W77INMBBHPRQXD5Q, seg_pos={0 0}, upload=440528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF Dec 26 13:26:45.660615572 (501h57m45.531698607s) [noise=true , address=145.79.227.30:4784 , piece_id=5JTNMIH7VY2PHJTAHAW4OBXCAHLOLIYUERA4W77INMBBHPRQXD5Q, seg_pos={0 0}, upload=440528]: Long tail cancellation due to optimal threshold Dec 26 13:26:45.660752162 (501h57m45.531835197s) [noise=true , address=77.90.31.17:4794 , piece_id=ZCCSOTOPZ4TB3V6R7T7C4ZN5ICF4H7CLPJGSQNJQ6LR47LT2ZLWA, seg_pos={0 0}, upload=440528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63): protocol: expected piece hash; context canceled; EOF Dec 26 13:26:45.660760227 (501h57m45.531843262s) [noise=true , address=77.90.31.17:4794 , piece_id=ZCCSOTOPZ4TB3V6R7T7C4ZN5ICF4H7CLPJGSQNJQ6LR47LT2ZLWA, seg_pos={0 0}, upload=440528]: Long tail cancellation due to optimal threshold Dec 26 13:26:45.660910863 (501h57m45.531993898s) [noise=true , address=77.90.31.14:4673 , piece_id=5WRLL4ABQX6X7OCOUJWU6LXNUBRBQZ6UOHPLA6ZEG5SRCXDLWLBQ, seg_pos={0 0}, upload=440528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh): protocol: read tcp 188.166.225.170:45006->77.90.31.14:4673: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 13:26:45.660924079 (501h57m45.532007114s) [noise=true , address=77.90.31.14:4673 , piece_id=5WRLL4ABQX6X7OCOUJWU6LXNUBRBQZ6UOHPLA6ZEG5SRCXDLWLBQ, seg_pos={0 0}, upload=440528]: Long tail cancellation due to optimal threshold Dec 26 13:26:45.660970287 (501h57m45.532053322s) [noise=true , address=64.49.13.169:4710 , piece_id=X6K6RY4XCSA2K2QN7YYMROCN5HSH2GAAPC2VQQW3O2322AOFTYNQ, seg_pos={0 0}, upload=440528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:51086->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:51086->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51086->64.49.13.169:4710: use of closed network connection Dec 26 13:26:45.660982530 (501h57m45.532065575s) [noise=true , address=64.49.13.169:4710 , piece_id=X6K6RY4XCSA2K2QN7YYMROCN5HSH2GAAPC2VQQW3O2322AOFTYNQ, seg_pos={0 0}, upload=440528]: Long tail cancellation due to optimal threshold Dec 26 13:26:45.661001787 (501h57m45.532084832s) [seg_pos={0 0}, upload=440528]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 13:26:45.661135261 (501h57m45.532218296s) [upload=440528]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 13:26:45.661140912 (501h57m45.532223947s) [upload=440528]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 13:26:45.960760851 (501h57m45.831843886s) [upload=440528]: Done uploading segments... Dec 26 13:26:45.960779797 (501h57m45.831862842s) [upload=440528]: Upload finished. err: <nil> Duration: 47.477243792s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 34 out of 60 is failed (threshold=6) [XXXXXX.XXXX.X..X..X....XXXXXX...XX.....X...XX.X.XXX.XXXXX..X] duration: 5.001453479s trace: https://jaeger.dev.storj.io/trace/17542e8e9ef9d6ac FAIL (2025-12-26 13:29:12 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001476665s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)