Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................XX....] duration: 7.67990176s trace: https://jaeger.dev.storj.io/trace/75ef8b5e6bc2819a FAIL (2025-12-27 14:01:28 UTC) orbiter warning: context deadline exceeded Duration: 7.679977704s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 37 out of 60 is failed (threshold=6) [XXX..XXXXX....XX....X.X.X.X.X..XXX.XX.XX.X.XX.X.XXXXXXXXXXX.] duration: 5.001352539s trace: https://jaeger.dev.storj.io/trace/15798ca0dffc45d0 FAIL (2025-12-27 14:01:17 UTC) orbiter warning: context deadline exceeded Duration: 5.001379129s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......X...X...X..........X.............XX..................] duration: 19.304361377s trace: https://jaeger.dev.storj.io/trace/3ed1e657d506c4ee FAIL (2025-12-27 14:00:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, Piece ID: HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ) context deadline exceeded; piecestore: (Node ID: 12wXqePK1kK7iBo6wHcbqkRmkED8LGJ3rzQNJAWa14cHYhx1oyB, Piece ID: M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA) context deadline exceeded; piecestore: (Node ID: 12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx, Piece ID: ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ) context deadline exceeded; piecestore: (Node ID: 12UjFkFn2UEvr57LDorcijDCvkBkkKitgZEFx6Yqfk5gMStQvJG, Piece ID: SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ) context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.16:4739 , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.319646322 (526h14m34.190729357s) [noise=true , address=77.90.31.21:4899 , piece_id=SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.319933248 (526h14m34.191016283s) [noise=true , address=45.91.116.12:4234 , piece_id=Y2FIVYLZBQAYBO2TDGIZ4RDQHQJ3WOQH565A7ESIKANQQTQDFQZQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320400588 (526h14m34.191483633s) [noise=true , address=145.79.227.29:4952 , piece_id=VXZLF4OGRIR3MYQMMLKLTD235A55F4YTHB3D4YCVCQUL4FZTSFJQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320407341 (526h14m34.191490386s) [noise=true , address=145.79.227.46:4621 , piece_id=XZ4MKAZAGQKVEJJGSZ5QGT4AT3MFXXNMXICVFCI6SP3E4EVEHZGQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320675030 (526h14m34.191758075s) [noise=true , address=45.91.116.10:4113 , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320802022 (526h14m34.191885067s) [noise=true , address=77.90.31.11:4569 , piece_id=YC2Q3HPJDVY3NEDZNC2Z4KVN6S6DDDGRARASF7KALFAOQ335A7HA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.321169643 (526h14m34.192252678s) [noise=true , address=77.90.31.19:4869 , piece_id=HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.321443483 (526h14m34.192526518s) [noise=true , address=45.91.116.14:4202 , piece_id=M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.321731813 (526h14m34.192814848s) [noise=true , address=77.90.31.18:4819 , piece_id=YS4DXTHLGCPFOKKLBA2DK2BVOAPTQEGVOPBSRCLFV26SVP6YFHGA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.322221776 (526h14m34.193304811s) [noise=true , address=77.90.31.15:4708 , piece_id=3NX6E54XMZXQAQQE27LTWUSEF5GAIF2XOPQFFAP7LOA2PL7KDL3Q, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.322547776 (526h14m34.193630811s) [noise=true , address=145.79.227.40:4079 , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.322908093 (526h14m34.193991138s) [noise=true , address=145.79.227.29:4975 , piece_id=DAMNBVP7DRSW5OAGYRRP4K5SNJ2WEUT7IAXO77KWRHLQEMBNERZQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.323267968 (526h14m34.194351003s) [noise=true , address=145.79.227.30:4787 , piece_id=OB2OTY3ZEHIUFYM6FY7HIS7HQ353Y2HL6XU4476HXOD26QVA4DPQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.323618165 (526h14m34.19470121s) [noise=true , address=64.49.13.169:4710 , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.324084893 (526h14m34.195167928s) [noise=true , address=193.149.164.143:4694 , piece_id=ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:35.783540136 (526h14m35.654623181s) [upload=461595]: Next returned nil segment Dec 27 13:43:35.783555856 (526h14m35.654638891s) [upload=461595]: Waiting for error group managing segments... Dec 27 13:43:36.192287250 (526h14m36.063370296s) [noise=true , address=145.79.227.29:4952 , piece_id=VXZLF4OGRIR3MYQMMLKLTD235A55F4YTHB3D4YCVCQUL4FZTSFJQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.200200289 (526h14m36.071283334s) [noise=true , address=145.79.227.46:4621 , piece_id=XZ4MKAZAGQKVEJJGSZ5QGT4AT3MFXXNMXICVFCI6SP3E4EVEHZGQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.238087532 (526h14m36.109170577s) [noise=true , address=145.79.227.29:4975 , piece_id=DAMNBVP7DRSW5OAGYRRP4K5SNJ2WEUT7IAXO77KWRHLQEMBNERZQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.379631346 (526h14m36.250714391s) [noise=true , address=77.90.31.19:4869 , piece_id=HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.382945286 (526h14m36.254028321s) [noise=true , address=145.79.227.34:4909 , piece_id=Z6RF5PSOGOZMOXPQYXHMXGFXIELVXXVU4NXXRTAQCIELQXLMVXBQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.400290103 (526h14m36.271373138s) [noise=true , address=77.90.31.18:4819 , piece_id=YS4DXTHLGCPFOKKLBA2DK2BVOAPTQEGVOPBSRCLFV26SVP6YFHGA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.436707208 (526h14m36.307790243s) [noise=true , address=45.91.116.14:4202 , piece_id=M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.487651475 (526h14m36.35873451s) [noise=true , address=45.91.116.12:4234 , piece_id=Y2FIVYLZBQAYBO2TDGIZ4RDQHQJ3WOQH565A7ESIKANQQTQDFQZQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.500850769 (526h14m36.371933814s) [noise=true , address=77.90.31.11:4569 , piece_id=YC2Q3HPJDVY3NEDZNC2Z4KVN6S6DDDGRARASF7KALFAOQ335A7HA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.518828551 (526h14m36.389911586s) [noise=true , address=77.90.31.15:4708 , piece_id=3NX6E54XMZXQAQQE27LTWUSEF5GAIF2XOPQFFAP7LOA2PL7KDL3Q, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.051607243 (526h14m36.922690289s) [noise=true , address=145.79.227.44:4228 , piece_id=6SAYEAOTTCOUFPU75C7YRIULWJGXL4PK2YKCKFB47EUBWMRQQQOQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.096826909 (526h14m36.967909954s) [noise=true , address=145.79.227.30:4787 , piece_id=OB2OTY3ZEHIUFYM6FY7HIS7HQ353Y2HL6XU4476HXOD26QVA4DPQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.140738995 (526h14m37.01182203s) [noise=true , address=77.90.31.21:4899 , piece_id=SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.323613990 (526h14m37.194697035s) [noise=true , address=193.149.164.143:4694 , piece_id=ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.323649618 (526h14m37.194732663s) [seg_pos={0 0}, upload=461595]: Segment reached optimal threshold of 14 pieces. Dec 27 13:43:37.323852865 (526h14m37.1949359s) [noise=true , address=45.91.116.10:4113 , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12puRMny4nbgv2by4tAbXcBBSgG2d6BfwcDDRDcFzEdjSVUJxtd): protocol: expected piece hash; context canceled; EOF Dec 27 13:43:37.323860349 (526h14m37.194943384s) [noise=true , address=45.91.116.10:4113 , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324005135 (526h14m37.1950882s) [noise=true , address=145.79.227.40:4079 , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RYKzn5zVSmxVbWwnBdX8oqtG1WE2MhPeEiWLf5dZeFAut6s2Y): protocol: expected piece hash; context canceled; EOF Dec 27 13:43:37.324022659 (526h14m37.195105704s) [noise=true , address=145.79.227.40:4079 , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324143378 (526h14m37.195226413s) [noise=true , address=77.90.31.16:4739 , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129B3N7yjJvwCzoVLTqWsPbcLQ5TQUyAzg1RCkbSAVKPWYTG8eb): protocol: expected piece hash; context canceled; EOF Dec 27 13:43:37.324154520 (526h14m37.195237555s) [noise=true , address=77.90.31.16:4739 , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324213442 (526h14m37.195296487s) [noise=true , address=64.49.13.169:4710 , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection Dec 27 13:43:37.324249360 (526h14m37.195332405s) [noise=true , address=64.49.13.169:4710 , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324275360 (526h14m37.195358405s) [seg_pos={0 0}, upload=461595]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 13:43:37.324374018 (526h14m37.195457053s) [upload=461595]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:43:37.324381291 (526h14m37.195464337s) [upload=461595]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:43:37.679036406 (526h14m37.550119451s) [upload=461595]: Done uploading segments... Dec 27 13:43:37.679056985 (526h14m37.55014003s) [upload=461595]: Upload finished. err: <nil> Duration: 19.304410871s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................XX...] duration: 18.484246449s trace: https://jaeger.dev.storj.io/trace/75ef8b5e6bc2819a FAIL (2025-12-27 14:00:41 UTC) orbiter warning: context deadline exceeded Duration: 18.484281346s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 38 out of 60 is failed (threshold=6) [XXXX..XXXXX....XX....X.X.X.X.X..XXX.XX.XX.X.XX.X.XXXXXXXXXXX] duration: 5.001330705s trace: https://jaeger.dev.storj.io/trace/15798ca0dffc45d0 FAIL (2025-12-27 14:00:21 UTC) orbiter warning: context deadline exceeded Duration: 5.001357135s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................................XX..] duration: 18.272737932s trace: https://jaeger.dev.storj.io/trace/75ef8b5e6bc2819a FAIL (2025-12-27 14:00:16 UTC) orbiter warning: context deadline exceeded Duration: 18.272756928s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..X.XXX....X...X...X...XX.......X....X..XX.X.XXXX..X..X.....] duration: 5.750859846s trace: https://jaeger.dev.storj.io/trace/61cbbcf7e1004bc3 FAIL (2025-12-27 14:00:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: WRCB4QABBNPHWTPXEEO4SLB7MR3MT467XFOZTAWCCWG67ZAAK4GA) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: 2X3DL3V773MFFXYVQYRIYKXIRU4JZI4KELFIXZS3PUTBPZVTGKRA) context deadline exceeded ======LOGS====== ing segments... Dec 27 13:55:39.506895724 (526h26m39.377978769s) [upload=461769]: Got next segment. Inline: false Dec 27 13:55:39.506937954 (526h26m39.378021s) [upload=461769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:55:39.506943695 (526h26m39.37802673s) [upload=461769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:55:39.746845108 (526h26m39.617928254s) [seg_pos={0 0}, upload=461769]: Begin upload segment... Dec 27 13:55:39.746972181 (526h26m39.618055226s) [seg_pos={0 0}, upload=461769]: Done begin upload segment. Dec 27 13:55:39.747039980 (526h26m39.618123025s) [noise=true , address=145.79.227.23:4546 , piece_id=SFSPRBAS5DZ7PGSPKN3F5RV4I2WPP6PXBQSZTIEVF2OZQIBY7PIQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.747055749 (526h26m39.618138794s) [noise=true , address=145.79.227.16:4912 , piece_id=2V6SNFMGJJF6PX7YA63PQUHCWEBEQQRXTIV5YP5IXPGOHL57DOWQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.747668517 (526h26m39.618751562s) [noise=true , address=145.79.227.28:4727 , piece_id=NSE2QHV2T375DF7JQO32WROCMIA3HUVBNBZZW4LCBCJ2K4CAL23Q, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.748154081 (526h26m39.619237136s) [noise=true , address=45.91.116.12:4225 , piece_id=NMARGNK6INEKMWLK2IQSTSMAVSE36E3T4WXMD6DEQMJKYTB6ZKXA, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.748539615 (526h26m39.61962266s) [noise=true , address=145.79.227.41:4085 , piece_id=4T7J4A56XHXQ7J5MP3PB5GUPL6ZQRYLTZ66IMD3GJVE4EGQY5JIA, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.748812355 (526h26m39.6198954s) [noise=true , address=77.90.31.12:4589 , piece_id=WRCB4QABBNPHWTPXEEO4SLB7MR3MT467XFOZTAWCCWG67ZAAK4GA, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.749151961 (526h26m39.620235036s) [noise=true , address=145.79.227.47:4267 , piece_id=TUZIXXBICDEZ64AI7XOVBBQRKM4WS7UVXOSFLGUXRZZXZJ5QFRZQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.749662393 (526h26m39.620745438s) [noise=true , address=145.79.227.33:4869 , piece_id=MCLA62SXY6NLKAE5AJJQS4EKPY6NH6CO4HNSEW4HU5PMAALAEDUA, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.750118381 (526h26m39.621201416s) [noise=true , address=145.79.227.27:4689 , piece_id=KCW5QW3O36SBVXA6BJV5DIRTS4HOYMQSAFRXCCCNHB3OPAXXFCFQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.750451205 (526h26m39.62153424s) [noise=true , address=145.79.227.37:4567 , piece_id=REVK53YXB7BT7T2IXTHM4NQBLC5LD2V6Q3AIX5I4O5Y2V5VQJDHQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.750760544 (526h26m39.62184358s) [noise=true , address=45.91.116.13:4153 , piece_id=VSAPME7HWTPXGRIHNZOOW7CHLHC55ELZUVRHNN7A7SHEVEEJEOYQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.751022002 (526h26m39.622105077s) [noise=true , address=145.79.227.29:4942 , piece_id=XQY6IDGGUO73AYWSQPJ5B2K724RWGVNDNRKPUBUKOGR7WPLBLWQQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.751446380 (526h26m39.622529415s) [noise=true , address=145.79.227.13:4804 , piece_id=LSHRLHBDQ5XYJCZ4XCOKY47EWU34SRRJWEORJS4PF4BTU7HA6DAA, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.751778563 (526h26m39.622861608s) [noise=true , address=77.90.31.11:4556 , piece_id=2X3DL3V773MFFXYVQYRIYKXIRU4JZI4KELFIXZS3PUTBPZVTGKRA, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.752151403 (526h26m39.623234438s) [noise=true , address=145.79.227.45:4231 , piece_id=ZI2FWBYUL4G7PCN5YZWAELM22AQP2O7Z6HML5NMGK5UJ3RSTEODQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.752460461 (526h26m39.623543496s) [noise=true , address=77.90.31.19:4842 , piece_id=OGWSBYPGFHPWZ5VJISQ7A3PSYN7GGS6QVYINEAWBGAP4ADKVMPAA, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:41.454144697 (526h26m41.325227742s) [upload=461769]: Next returned nil segment Dec 27 13:55:41.454167932 (526h26m41.325250977s) [upload=461769]: Waiting for error group managing segments... Dec 27 13:55:41.738963230 (526h26m41.610046265s) [noise=true , address=145.79.227.37:4567 , piece_id=REVK53YXB7BT7T2IXTHM4NQBLC5LD2V6Q3AIX5I4O5Y2V5VQJDHQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.742975230 (526h26m41.614058265s) [noise=true , address=145.79.227.27:4689 , piece_id=KCW5QW3O36SBVXA6BJV5DIRTS4HOYMQSAFRXCCCNHB3OPAXXFCFQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.751149826 (526h26m41.622232871s) [noise=true , address=145.79.227.41:4085 , piece_id=4T7J4A56XHXQ7J5MP3PB5GUPL6ZQRYLTZ66IMD3GJVE4EGQY5JIA, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.751779916 (526h26m41.622862951s) [noise=true , address=145.79.227.45:4231 , piece_id=ZI2FWBYUL4G7PCN5YZWAELM22AQP2O7Z6HML5NMGK5UJ3RSTEODQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.755286142 (526h26m41.626369177s) [noise=true , address=145.79.227.16:4912 , piece_id=2V6SNFMGJJF6PX7YA63PQUHCWEBEQQRXTIV5YP5IXPGOHL57DOWQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.763271658 (526h26m41.634354693s) [noise=true , address=145.79.227.13:4804 , piece_id=LSHRLHBDQ5XYJCZ4XCOKY47EWU34SRRJWEORJS4PF4BTU7HA6DAA, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.764016025 (526h26m41.63509906s) [noise=true , address=145.79.227.28:4727 , piece_id=NSE2QHV2T375DF7JQO32WROCMIA3HUVBNBZZW4LCBCJ2K4CAL23Q, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.764095897 (526h26m41.635178932s) [noise=true , address=145.79.227.29:4942 , piece_id=XQY6IDGGUO73AYWSQPJ5B2K724RWGVNDNRKPUBUKOGR7WPLBLWQQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.778988115 (526h26m41.65007116s) [noise=true , address=145.79.227.47:4267 , piece_id=TUZIXXBICDEZ64AI7XOVBBQRKM4WS7UVXOSFLGUXRZZXZJ5QFRZQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.783665982 (526h26m41.654749027s) [noise=true , address=145.79.227.23:4546 , piece_id=SFSPRBAS5DZ7PGSPKN3F5RV4I2WPP6PXBQSZTIEVF2OZQIBY7PIQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:42.700160844 (526h26m42.571243879s) [noise=true , address=77.90.31.11:4556 , piece_id=2X3DL3V773MFFXYVQYRIYKXIRU4JZI4KELFIXZS3PUTBPZVTGKRA, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:42.928495633 (526h26m42.799578678s) [noise=true , address=77.90.31.19:4842 , piece_id=OGWSBYPGFHPWZ5VJISQ7A3PSYN7GGS6QVYINEAWBGAP4ADKVMPAA, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:43.041556590 (526h26m42.912639625s) [noise=true , address=77.90.31.12:4589 , piece_id=WRCB4QABBNPHWTPXEEO4SLB7MR3MT467XFOZTAWCCWG67ZAAK4GA, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:43.041576267 (526h26m42.912659302s) [seg_pos={0 0}, upload=461769]: Segment reached optimal threshold of 13 pieces. Dec 27 13:55:43.041779755 (526h26m42.91286279s) [noise=true , address=145.79.227.33:4869 , piece_id=MCLA62SXY6NLKAE5AJJQS4EKPY6NH6CO4HNSEW4HU5PMAALAEDUA, seg_pos={0 0}, upload=461769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 27 13:55:43.041785526 (526h26m42.912868561s) [noise=true , address=145.79.227.33:4869 , piece_id=MCLA62SXY6NLKAE5AJJQS4EKPY6NH6CO4HNSEW4HU5PMAALAEDUA, seg_pos={0 0}, upload=461769]: Long tail cancellation due to optimal threshold Dec 27 13:55:43.042105304 (526h26m42.91318837s) [noise=true , address=45.91.116.13:4153 , piece_id=VSAPME7HWTPXGRIHNZOOW7CHLHC55ELZUVRHNN7A7SHEVEEJEOYQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:36608->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:36608->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36608->45.91.116.13:4153: use of closed network connection Dec 27 13:55:43.042131284 (526h26m42.913214319s) [noise=true , address=45.91.116.13:4153 , piece_id=VSAPME7HWTPXGRIHNZOOW7CHLHC55ELZUVRHNN7A7SHEVEEJEOYQ, seg_pos={0 0}, upload=461769]: Long tail cancellation due to optimal threshold Dec 27 13:55:43.042255090 (526h26m42.913338125s) [noise=true , address=45.91.116.12:4225 , piece_id=NMARGNK6INEKMWLK2IQSTSMAVSE36E3T4WXMD6DEQMJKYTB6ZKXA, seg_pos={0 0}, upload=461769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:47484->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:47484->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47484->45.91.116.12:4225: use of closed network connection Dec 27 13:55:43.042264498 (526h26m42.913347533s) [noise=true , address=45.91.116.12:4225 , piece_id=NMARGNK6INEKMWLK2IQSTSMAVSE36E3T4WXMD6DEQMJKYTB6ZKXA, seg_pos={0 0}, upload=461769]: Long tail cancellation due to optimal threshold Dec 27 13:55:43.042283343 (526h26m42.913366388s) [seg_pos={0 0}, upload=461769]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:55:43.042387552 (526h26m42.913470597s) [upload=461769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:55:43.042394376 (526h26m42.913477411s) [upload=461769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:55:43.323639824 (526h26m43.194722859s) [upload=461769]: Done uploading segments... Dec 27 13:55:43.323658681 (526h26m43.194741716s) [upload=461769]: Upload finished. err: <nil> Duration: 5.750892638s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [........X...X...X..........X.............XX.................] duration: 35.621614485s trace: https://jaeger.dev.storj.io/trace/3ed1e657d506c4ee FAIL (2025-12-27 13:59:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, Piece ID: HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ) context deadline exceeded; piecestore: (Node ID: 12wXqePK1kK7iBo6wHcbqkRmkED8LGJ3rzQNJAWa14cHYhx1oyB, Piece ID: M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA) context deadline exceeded; piecestore: (Node ID: 12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx, Piece ID: ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ) context deadline exceeded; piecestore: (Node ID: 12UjFkFn2UEvr57LDorcijDCvkBkkKitgZEFx6Yqfk5gMStQvJG, Piece ID: SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ) context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.16:4739 , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.319646322 (526h14m34.190729357s) [noise=true , address=77.90.31.21:4899 , piece_id=SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.319933248 (526h14m34.191016283s) [noise=true , address=45.91.116.12:4234 , piece_id=Y2FIVYLZBQAYBO2TDGIZ4RDQHQJ3WOQH565A7ESIKANQQTQDFQZQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320400588 (526h14m34.191483633s) [noise=true , address=145.79.227.29:4952 , piece_id=VXZLF4OGRIR3MYQMMLKLTD235A55F4YTHB3D4YCVCQUL4FZTSFJQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320407341 (526h14m34.191490386s) [noise=true , address=145.79.227.46:4621 , piece_id=XZ4MKAZAGQKVEJJGSZ5QGT4AT3MFXXNMXICVFCI6SP3E4EVEHZGQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320675030 (526h14m34.191758075s) [noise=true , address=45.91.116.10:4113 , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320802022 (526h14m34.191885067s) [noise=true , address=77.90.31.11:4569 , piece_id=YC2Q3HPJDVY3NEDZNC2Z4KVN6S6DDDGRARASF7KALFAOQ335A7HA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.321169643 (526h14m34.192252678s) [noise=true , address=77.90.31.19:4869 , piece_id=HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.321443483 (526h14m34.192526518s) [noise=true , address=45.91.116.14:4202 , piece_id=M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.321731813 (526h14m34.192814848s) [noise=true , address=77.90.31.18:4819 , piece_id=YS4DXTHLGCPFOKKLBA2DK2BVOAPTQEGVOPBSRCLFV26SVP6YFHGA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.322221776 (526h14m34.193304811s) [noise=true , address=77.90.31.15:4708 , piece_id=3NX6E54XMZXQAQQE27LTWUSEF5GAIF2XOPQFFAP7LOA2PL7KDL3Q, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.322547776 (526h14m34.193630811s) [noise=true , address=145.79.227.40:4079 , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.322908093 (526h14m34.193991138s) [noise=true , address=145.79.227.29:4975 , piece_id=DAMNBVP7DRSW5OAGYRRP4K5SNJ2WEUT7IAXO77KWRHLQEMBNERZQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.323267968 (526h14m34.194351003s) [noise=true , address=145.79.227.30:4787 , piece_id=OB2OTY3ZEHIUFYM6FY7HIS7HQ353Y2HL6XU4476HXOD26QVA4DPQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.323618165 (526h14m34.19470121s) [noise=true , address=64.49.13.169:4710 , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.324084893 (526h14m34.195167928s) [noise=true , address=193.149.164.143:4694 , piece_id=ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:35.783540136 (526h14m35.654623181s) [upload=461595]: Next returned nil segment Dec 27 13:43:35.783555856 (526h14m35.654638891s) [upload=461595]: Waiting for error group managing segments... Dec 27 13:43:36.192287250 (526h14m36.063370296s) [noise=true , address=145.79.227.29:4952 , piece_id=VXZLF4OGRIR3MYQMMLKLTD235A55F4YTHB3D4YCVCQUL4FZTSFJQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.200200289 (526h14m36.071283334s) [noise=true , address=145.79.227.46:4621 , piece_id=XZ4MKAZAGQKVEJJGSZ5QGT4AT3MFXXNMXICVFCI6SP3E4EVEHZGQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.238087532 (526h14m36.109170577s) [noise=true , address=145.79.227.29:4975 , piece_id=DAMNBVP7DRSW5OAGYRRP4K5SNJ2WEUT7IAXO77KWRHLQEMBNERZQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.379631346 (526h14m36.250714391s) [noise=true , address=77.90.31.19:4869 , piece_id=HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.382945286 (526h14m36.254028321s) [noise=true , address=145.79.227.34:4909 , piece_id=Z6RF5PSOGOZMOXPQYXHMXGFXIELVXXVU4NXXRTAQCIELQXLMVXBQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.400290103 (526h14m36.271373138s) [noise=true , address=77.90.31.18:4819 , piece_id=YS4DXTHLGCPFOKKLBA2DK2BVOAPTQEGVOPBSRCLFV26SVP6YFHGA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.436707208 (526h14m36.307790243s) [noise=true , address=45.91.116.14:4202 , piece_id=M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.487651475 (526h14m36.35873451s) [noise=true , address=45.91.116.12:4234 , piece_id=Y2FIVYLZBQAYBO2TDGIZ4RDQHQJ3WOQH565A7ESIKANQQTQDFQZQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.500850769 (526h14m36.371933814s) [noise=true , address=77.90.31.11:4569 , piece_id=YC2Q3HPJDVY3NEDZNC2Z4KVN6S6DDDGRARASF7KALFAOQ335A7HA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.518828551 (526h14m36.389911586s) [noise=true , address=77.90.31.15:4708 , piece_id=3NX6E54XMZXQAQQE27LTWUSEF5GAIF2XOPQFFAP7LOA2PL7KDL3Q, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.051607243 (526h14m36.922690289s) [noise=true , address=145.79.227.44:4228 , piece_id=6SAYEAOTTCOUFPU75C7YRIULWJGXL4PK2YKCKFB47EUBWMRQQQOQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.096826909 (526h14m36.967909954s) [noise=true , address=145.79.227.30:4787 , piece_id=OB2OTY3ZEHIUFYM6FY7HIS7HQ353Y2HL6XU4476HXOD26QVA4DPQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.140738995 (526h14m37.01182203s) [noise=true , address=77.90.31.21:4899 , piece_id=SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.323613990 (526h14m37.194697035s) [noise=true , address=193.149.164.143:4694 , piece_id=ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.323649618 (526h14m37.194732663s) [seg_pos={0 0}, upload=461595]: Segment reached optimal threshold of 14 pieces. Dec 27 13:43:37.323852865 (526h14m37.1949359s) [noise=true , address=45.91.116.10:4113 , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12puRMny4nbgv2by4tAbXcBBSgG2d6BfwcDDRDcFzEdjSVUJxtd): protocol: expected piece hash; context canceled; EOF Dec 27 13:43:37.323860349 (526h14m37.194943384s) [noise=true , address=45.91.116.10:4113 , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324005135 (526h14m37.1950882s) [noise=true , address=145.79.227.40:4079 , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RYKzn5zVSmxVbWwnBdX8oqtG1WE2MhPeEiWLf5dZeFAut6s2Y): protocol: expected piece hash; context canceled; EOF Dec 27 13:43:37.324022659 (526h14m37.195105704s) [noise=true , address=145.79.227.40:4079 , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324143378 (526h14m37.195226413s) [noise=true , address=77.90.31.16:4739 , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129B3N7yjJvwCzoVLTqWsPbcLQ5TQUyAzg1RCkbSAVKPWYTG8eb): protocol: expected piece hash; context canceled; EOF Dec 27 13:43:37.324154520 (526h14m37.195237555s) [noise=true , address=77.90.31.16:4739 , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324213442 (526h14m37.195296487s) [noise=true , address=64.49.13.169:4710 , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection Dec 27 13:43:37.324249360 (526h14m37.195332405s) [noise=true , address=64.49.13.169:4710 , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324275360 (526h14m37.195358405s) [seg_pos={0 0}, upload=461595]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 13:43:37.324374018 (526h14m37.195457053s) [upload=461595]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:43:37.324381291 (526h14m37.195464337s) [upload=461595]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:43:37.679036406 (526h14m37.550119451s) [upload=461595]: Done uploading segments... Dec 27 13:43:37.679056985 (526h14m37.55014003s) [upload=461595]: Upload finished. err: <nil> Duration: 35.621653609s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................XX.] duration: 14.200030414s trace: https://jaeger.dev.storj.io/trace/75ef8b5e6bc2819a FAIL (2025-12-27 13:59:42 UTC) orbiter warning: context deadline exceeded Duration: 14.200081361s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........X...X...X..........X.............XX................] duration: 43.172061397s trace: https://jaeger.dev.storj.io/trace/3ed1e657d506c4ee FAIL (2025-12-27 13:59:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, Piece ID: HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ) context deadline exceeded; piecestore: (Node ID: 12wXqePK1kK7iBo6wHcbqkRmkED8LGJ3rzQNJAWa14cHYhx1oyB, Piece ID: M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA) context deadline exceeded; piecestore: (Node ID: 12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx, Piece ID: ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ) context deadline exceeded; piecestore: (Node ID: 12UjFkFn2UEvr57LDorcijDCvkBkkKitgZEFx6Yqfk5gMStQvJG, Piece ID: SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ) context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.16:4739 , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.319646322 (526h14m34.190729357s) [noise=true , address=77.90.31.21:4899 , piece_id=SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.319933248 (526h14m34.191016283s) [noise=true , address=45.91.116.12:4234 , piece_id=Y2FIVYLZBQAYBO2TDGIZ4RDQHQJ3WOQH565A7ESIKANQQTQDFQZQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320400588 (526h14m34.191483633s) [noise=true , address=145.79.227.29:4952 , piece_id=VXZLF4OGRIR3MYQMMLKLTD235A55F4YTHB3D4YCVCQUL4FZTSFJQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320407341 (526h14m34.191490386s) [noise=true , address=145.79.227.46:4621 , piece_id=XZ4MKAZAGQKVEJJGSZ5QGT4AT3MFXXNMXICVFCI6SP3E4EVEHZGQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320675030 (526h14m34.191758075s) [noise=true , address=45.91.116.10:4113 , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320802022 (526h14m34.191885067s) [noise=true , address=77.90.31.11:4569 , piece_id=YC2Q3HPJDVY3NEDZNC2Z4KVN6S6DDDGRARASF7KALFAOQ335A7HA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.321169643 (526h14m34.192252678s) [noise=true , address=77.90.31.19:4869 , piece_id=HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.321443483 (526h14m34.192526518s) [noise=true , address=45.91.116.14:4202 , piece_id=M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.321731813 (526h14m34.192814848s) [noise=true , address=77.90.31.18:4819 , piece_id=YS4DXTHLGCPFOKKLBA2DK2BVOAPTQEGVOPBSRCLFV26SVP6YFHGA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.322221776 (526h14m34.193304811s) [noise=true , address=77.90.31.15:4708 , piece_id=3NX6E54XMZXQAQQE27LTWUSEF5GAIF2XOPQFFAP7LOA2PL7KDL3Q, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.322547776 (526h14m34.193630811s) [noise=true , address=145.79.227.40:4079 , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.322908093 (526h14m34.193991138s) [noise=true , address=145.79.227.29:4975 , piece_id=DAMNBVP7DRSW5OAGYRRP4K5SNJ2WEUT7IAXO77KWRHLQEMBNERZQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.323267968 (526h14m34.194351003s) [noise=true , address=145.79.227.30:4787 , piece_id=OB2OTY3ZEHIUFYM6FY7HIS7HQ353Y2HL6XU4476HXOD26QVA4DPQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.323618165 (526h14m34.19470121s) [noise=true , address=64.49.13.169:4710 , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.324084893 (526h14m34.195167928s) [noise=true , address=193.149.164.143:4694 , piece_id=ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:35.783540136 (526h14m35.654623181s) [upload=461595]: Next returned nil segment Dec 27 13:43:35.783555856 (526h14m35.654638891s) [upload=461595]: Waiting for error group managing segments... Dec 27 13:43:36.192287250 (526h14m36.063370296s) [noise=true , address=145.79.227.29:4952 , piece_id=VXZLF4OGRIR3MYQMMLKLTD235A55F4YTHB3D4YCVCQUL4FZTSFJQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.200200289 (526h14m36.071283334s) [noise=true , address=145.79.227.46:4621 , piece_id=XZ4MKAZAGQKVEJJGSZ5QGT4AT3MFXXNMXICVFCI6SP3E4EVEHZGQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.238087532 (526h14m36.109170577s) [noise=true , address=145.79.227.29:4975 , piece_id=DAMNBVP7DRSW5OAGYRRP4K5SNJ2WEUT7IAXO77KWRHLQEMBNERZQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.379631346 (526h14m36.250714391s) [noise=true , address=77.90.31.19:4869 , piece_id=HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.382945286 (526h14m36.254028321s) [noise=true , address=145.79.227.34:4909 , piece_id=Z6RF5PSOGOZMOXPQYXHMXGFXIELVXXVU4NXXRTAQCIELQXLMVXBQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.400290103 (526h14m36.271373138s) [noise=true , address=77.90.31.18:4819 , piece_id=YS4DXTHLGCPFOKKLBA2DK2BVOAPTQEGVOPBSRCLFV26SVP6YFHGA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.436707208 (526h14m36.307790243s) [noise=true , address=45.91.116.14:4202 , piece_id=M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.487651475 (526h14m36.35873451s) [noise=true , address=45.91.116.12:4234 , piece_id=Y2FIVYLZBQAYBO2TDGIZ4RDQHQJ3WOQH565A7ESIKANQQTQDFQZQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.500850769 (526h14m36.371933814s) [noise=true , address=77.90.31.11:4569 , piece_id=YC2Q3HPJDVY3NEDZNC2Z4KVN6S6DDDGRARASF7KALFAOQ335A7HA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.518828551 (526h14m36.389911586s) [noise=true , address=77.90.31.15:4708 , piece_id=3NX6E54XMZXQAQQE27LTWUSEF5GAIF2XOPQFFAP7LOA2PL7KDL3Q, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.051607243 (526h14m36.922690289s) [noise=true , address=145.79.227.44:4228 , piece_id=6SAYEAOTTCOUFPU75C7YRIULWJGXL4PK2YKCKFB47EUBWMRQQQOQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.096826909 (526h14m36.967909954s) [noise=true , address=145.79.227.30:4787 , piece_id=OB2OTY3ZEHIUFYM6FY7HIS7HQ353Y2HL6XU4476HXOD26QVA4DPQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.140738995 (526h14m37.01182203s) [noise=true , address=77.90.31.21:4899 , piece_id=SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.323613990 (526h14m37.194697035s) [noise=true , address=193.149.164.143:4694 , piece_id=ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.323649618 (526h14m37.194732663s) [seg_pos={0 0}, upload=461595]: Segment reached optimal threshold of 14 pieces. Dec 27 13:43:37.323852865 (526h14m37.1949359s) [noise=true , address=45.91.116.10:4113 , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12puRMny4nbgv2by4tAbXcBBSgG2d6BfwcDDRDcFzEdjSVUJxtd): protocol: expected piece hash; context canceled; EOF Dec 27 13:43:37.323860349 (526h14m37.194943384s) [noise=true , address=45.91.116.10:4113 , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324005135 (526h14m37.1950882s) [noise=true , address=145.79.227.40:4079 , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RYKzn5zVSmxVbWwnBdX8oqtG1WE2MhPeEiWLf5dZeFAut6s2Y): protocol: expected piece hash; context canceled; EOF Dec 27 13:43:37.324022659 (526h14m37.195105704s) [noise=true , address=145.79.227.40:4079 , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324143378 (526h14m37.195226413s) [noise=true , address=77.90.31.16:4739 , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129B3N7yjJvwCzoVLTqWsPbcLQ5TQUyAzg1RCkbSAVKPWYTG8eb): protocol: expected piece hash; context canceled; EOF Dec 27 13:43:37.324154520 (526h14m37.195237555s) [noise=true , address=77.90.31.16:4739 , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324213442 (526h14m37.195296487s) [noise=true , address=64.49.13.169:4710 , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection Dec 27 13:43:37.324249360 (526h14m37.195332405s) [noise=true , address=64.49.13.169:4710 , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324275360 (526h14m37.195358405s) [seg_pos={0 0}, upload=461595]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 13:43:37.324374018 (526h14m37.195457053s) [upload=461595]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:43:37.324381291 (526h14m37.195464337s) [upload=461595]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:43:37.679036406 (526h14m37.550119451s) [upload=461595]: Done uploading segments... Dec 27 13:43:37.679056985 (526h14m37.55014003s) [upload=461595]: Upload finished. err: <nil> Duration: 43.172112814s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 38 out of 60 is failed (threshold=6) [XXXXX..XXXXX....XX....X.X.X.X.X..XXX.XX.XX.X.XX.X.XXXXXXXXXX] duration: 5.001566565s trace: https://jaeger.dev.storj.io/trace/6da20589251cb5ed FAIL (2025-12-27 13:59:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001724867s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 30.001389472s trace: https://jaeger.dev.storj.io/trace/75ef8b5e6bc2819a FAIL (2025-12-27 13:58:54 UTC) orbiter warning: context deadline exceeded Duration: 30.001412165s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [...X.XXX....X...X...X...XX.......X....X..XX.X.XXXX..X..X....] duration: 5.666307133s trace: https://jaeger.dev.storj.io/trace/61cbbcf7e1004bc3 FAIL (2025-12-27 13:58:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: WRCB4QABBNPHWTPXEEO4SLB7MR3MT467XFOZTAWCCWG67ZAAK4GA) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: 2X3DL3V773MFFXYVQYRIYKXIRU4JZI4KELFIXZS3PUTBPZVTGKRA) context deadline exceeded ======LOGS====== ing segments... Dec 27 13:55:39.506895724 (526h26m39.377978769s) [upload=461769]: Got next segment. Inline: false Dec 27 13:55:39.506937954 (526h26m39.378021s) [upload=461769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:55:39.506943695 (526h26m39.37802673s) [upload=461769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:55:39.746845108 (526h26m39.617928254s) [seg_pos={0 0}, upload=461769]: Begin upload segment... Dec 27 13:55:39.746972181 (526h26m39.618055226s) [seg_pos={0 0}, upload=461769]: Done begin upload segment. Dec 27 13:55:39.747039980 (526h26m39.618123025s) [noise=true , address=145.79.227.23:4546 , piece_id=SFSPRBAS5DZ7PGSPKN3F5RV4I2WPP6PXBQSZTIEVF2OZQIBY7PIQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.747055749 (526h26m39.618138794s) [noise=true , address=145.79.227.16:4912 , piece_id=2V6SNFMGJJF6PX7YA63PQUHCWEBEQQRXTIV5YP5IXPGOHL57DOWQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.747668517 (526h26m39.618751562s) [noise=true , address=145.79.227.28:4727 , piece_id=NSE2QHV2T375DF7JQO32WROCMIA3HUVBNBZZW4LCBCJ2K4CAL23Q, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.748154081 (526h26m39.619237136s) [noise=true , address=45.91.116.12:4225 , piece_id=NMARGNK6INEKMWLK2IQSTSMAVSE36E3T4WXMD6DEQMJKYTB6ZKXA, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.748539615 (526h26m39.61962266s) [noise=true , address=145.79.227.41:4085 , piece_id=4T7J4A56XHXQ7J5MP3PB5GUPL6ZQRYLTZ66IMD3GJVE4EGQY5JIA, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.748812355 (526h26m39.6198954s) [noise=true , address=77.90.31.12:4589 , piece_id=WRCB4QABBNPHWTPXEEO4SLB7MR3MT467XFOZTAWCCWG67ZAAK4GA, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.749151961 (526h26m39.620235036s) [noise=true , address=145.79.227.47:4267 , piece_id=TUZIXXBICDEZ64AI7XOVBBQRKM4WS7UVXOSFLGUXRZZXZJ5QFRZQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.749662393 (526h26m39.620745438s) [noise=true , address=145.79.227.33:4869 , piece_id=MCLA62SXY6NLKAE5AJJQS4EKPY6NH6CO4HNSEW4HU5PMAALAEDUA, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.750118381 (526h26m39.621201416s) [noise=true , address=145.79.227.27:4689 , piece_id=KCW5QW3O36SBVXA6BJV5DIRTS4HOYMQSAFRXCCCNHB3OPAXXFCFQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.750451205 (526h26m39.62153424s) [noise=true , address=145.79.227.37:4567 , piece_id=REVK53YXB7BT7T2IXTHM4NQBLC5LD2V6Q3AIX5I4O5Y2V5VQJDHQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.750760544 (526h26m39.62184358s) [noise=true , address=45.91.116.13:4153 , piece_id=VSAPME7HWTPXGRIHNZOOW7CHLHC55ELZUVRHNN7A7SHEVEEJEOYQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.751022002 (526h26m39.622105077s) [noise=true , address=145.79.227.29:4942 , piece_id=XQY6IDGGUO73AYWSQPJ5B2K724RWGVNDNRKPUBUKOGR7WPLBLWQQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.751446380 (526h26m39.622529415s) [noise=true , address=145.79.227.13:4804 , piece_id=LSHRLHBDQ5XYJCZ4XCOKY47EWU34SRRJWEORJS4PF4BTU7HA6DAA, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.751778563 (526h26m39.622861608s) [noise=true , address=77.90.31.11:4556 , piece_id=2X3DL3V773MFFXYVQYRIYKXIRU4JZI4KELFIXZS3PUTBPZVTGKRA, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.752151403 (526h26m39.623234438s) [noise=true , address=145.79.227.45:4231 , piece_id=ZI2FWBYUL4G7PCN5YZWAELM22AQP2O7Z6HML5NMGK5UJ3RSTEODQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.752460461 (526h26m39.623543496s) [noise=true , address=77.90.31.19:4842 , piece_id=OGWSBYPGFHPWZ5VJISQ7A3PSYN7GGS6QVYINEAWBGAP4ADKVMPAA, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:41.454144697 (526h26m41.325227742s) [upload=461769]: Next returned nil segment Dec 27 13:55:41.454167932 (526h26m41.325250977s) [upload=461769]: Waiting for error group managing segments... Dec 27 13:55:41.738963230 (526h26m41.610046265s) [noise=true , address=145.79.227.37:4567 , piece_id=REVK53YXB7BT7T2IXTHM4NQBLC5LD2V6Q3AIX5I4O5Y2V5VQJDHQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.742975230 (526h26m41.614058265s) [noise=true , address=145.79.227.27:4689 , piece_id=KCW5QW3O36SBVXA6BJV5DIRTS4HOYMQSAFRXCCCNHB3OPAXXFCFQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.751149826 (526h26m41.622232871s) [noise=true , address=145.79.227.41:4085 , piece_id=4T7J4A56XHXQ7J5MP3PB5GUPL6ZQRYLTZ66IMD3GJVE4EGQY5JIA, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.751779916 (526h26m41.622862951s) [noise=true , address=145.79.227.45:4231 , piece_id=ZI2FWBYUL4G7PCN5YZWAELM22AQP2O7Z6HML5NMGK5UJ3RSTEODQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.755286142 (526h26m41.626369177s) [noise=true , address=145.79.227.16:4912 , piece_id=2V6SNFMGJJF6PX7YA63PQUHCWEBEQQRXTIV5YP5IXPGOHL57DOWQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.763271658 (526h26m41.634354693s) [noise=true , address=145.79.227.13:4804 , piece_id=LSHRLHBDQ5XYJCZ4XCOKY47EWU34SRRJWEORJS4PF4BTU7HA6DAA, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.764016025 (526h26m41.63509906s) [noise=true , address=145.79.227.28:4727 , piece_id=NSE2QHV2T375DF7JQO32WROCMIA3HUVBNBZZW4LCBCJ2K4CAL23Q, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.764095897 (526h26m41.635178932s) [noise=true , address=145.79.227.29:4942 , piece_id=XQY6IDGGUO73AYWSQPJ5B2K724RWGVNDNRKPUBUKOGR7WPLBLWQQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.778988115 (526h26m41.65007116s) [noise=true , address=145.79.227.47:4267 , piece_id=TUZIXXBICDEZ64AI7XOVBBQRKM4WS7UVXOSFLGUXRZZXZJ5QFRZQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.783665982 (526h26m41.654749027s) [noise=true , address=145.79.227.23:4546 , piece_id=SFSPRBAS5DZ7PGSPKN3F5RV4I2WPP6PXBQSZTIEVF2OZQIBY7PIQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:42.700160844 (526h26m42.571243879s) [noise=true , address=77.90.31.11:4556 , piece_id=2X3DL3V773MFFXYVQYRIYKXIRU4JZI4KELFIXZS3PUTBPZVTGKRA, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:42.928495633 (526h26m42.799578678s) [noise=true , address=77.90.31.19:4842 , piece_id=OGWSBYPGFHPWZ5VJISQ7A3PSYN7GGS6QVYINEAWBGAP4ADKVMPAA, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:43.041556590 (526h26m42.912639625s) [noise=true , address=77.90.31.12:4589 , piece_id=WRCB4QABBNPHWTPXEEO4SLB7MR3MT467XFOZTAWCCWG67ZAAK4GA, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:43.041576267 (526h26m42.912659302s) [seg_pos={0 0}, upload=461769]: Segment reached optimal threshold of 13 pieces. Dec 27 13:55:43.041779755 (526h26m42.91286279s) [noise=true , address=145.79.227.33:4869 , piece_id=MCLA62SXY6NLKAE5AJJQS4EKPY6NH6CO4HNSEW4HU5PMAALAEDUA, seg_pos={0 0}, upload=461769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 27 13:55:43.041785526 (526h26m42.912868561s) [noise=true , address=145.79.227.33:4869 , piece_id=MCLA62SXY6NLKAE5AJJQS4EKPY6NH6CO4HNSEW4HU5PMAALAEDUA, seg_pos={0 0}, upload=461769]: Long tail cancellation due to optimal threshold Dec 27 13:55:43.042105304 (526h26m42.91318837s) [noise=true , address=45.91.116.13:4153 , piece_id=VSAPME7HWTPXGRIHNZOOW7CHLHC55ELZUVRHNN7A7SHEVEEJEOYQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:36608->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:36608->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36608->45.91.116.13:4153: use of closed network connection Dec 27 13:55:43.042131284 (526h26m42.913214319s) [noise=true , address=45.91.116.13:4153 , piece_id=VSAPME7HWTPXGRIHNZOOW7CHLHC55ELZUVRHNN7A7SHEVEEJEOYQ, seg_pos={0 0}, upload=461769]: Long tail cancellation due to optimal threshold Dec 27 13:55:43.042255090 (526h26m42.913338125s) [noise=true , address=45.91.116.12:4225 , piece_id=NMARGNK6INEKMWLK2IQSTSMAVSE36E3T4WXMD6DEQMJKYTB6ZKXA, seg_pos={0 0}, upload=461769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:47484->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:47484->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47484->45.91.116.12:4225: use of closed network connection Dec 27 13:55:43.042264498 (526h26m42.913347533s) [noise=true , address=45.91.116.12:4225 , piece_id=NMARGNK6INEKMWLK2IQSTSMAVSE36E3T4WXMD6DEQMJKYTB6ZKXA, seg_pos={0 0}, upload=461769]: Long tail cancellation due to optimal threshold Dec 27 13:55:43.042283343 (526h26m42.913366388s) [seg_pos={0 0}, upload=461769]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:55:43.042387552 (526h26m42.913470597s) [upload=461769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:55:43.042394376 (526h26m42.913477411s) [upload=461769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:55:43.323639824 (526h26m43.194722859s) [upload=461769]: Done uploading segments... Dec 27 13:55:43.323658681 (526h26m43.194741716s) [upload=461769]: Upload finished. err: <nil> Duration: 5.666334524s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........X...X...X..........X.............XX...............] duration: 52.139158449s trace: https://jaeger.dev.storj.io/trace/3ed1e657d506c4ee FAIL (2025-12-27 13:58:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, Piece ID: HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ) context deadline exceeded; piecestore: (Node ID: 12wXqePK1kK7iBo6wHcbqkRmkED8LGJ3rzQNJAWa14cHYhx1oyB, Piece ID: M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA) context deadline exceeded; piecestore: (Node ID: 12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx, Piece ID: ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ) context deadline exceeded; piecestore: (Node ID: 12UjFkFn2UEvr57LDorcijDCvkBkkKitgZEFx6Yqfk5gMStQvJG, Piece ID: SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ) context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.16:4739 , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.319646322 (526h14m34.190729357s) [noise=true , address=77.90.31.21:4899 , piece_id=SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.319933248 (526h14m34.191016283s) [noise=true , address=45.91.116.12:4234 , piece_id=Y2FIVYLZBQAYBO2TDGIZ4RDQHQJ3WOQH565A7ESIKANQQTQDFQZQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320400588 (526h14m34.191483633s) [noise=true , address=145.79.227.29:4952 , piece_id=VXZLF4OGRIR3MYQMMLKLTD235A55F4YTHB3D4YCVCQUL4FZTSFJQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320407341 (526h14m34.191490386s) [noise=true , address=145.79.227.46:4621 , piece_id=XZ4MKAZAGQKVEJJGSZ5QGT4AT3MFXXNMXICVFCI6SP3E4EVEHZGQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320675030 (526h14m34.191758075s) [noise=true , address=45.91.116.10:4113 , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320802022 (526h14m34.191885067s) [noise=true , address=77.90.31.11:4569 , piece_id=YC2Q3HPJDVY3NEDZNC2Z4KVN6S6DDDGRARASF7KALFAOQ335A7HA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.321169643 (526h14m34.192252678s) [noise=true , address=77.90.31.19:4869 , piece_id=HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.321443483 (526h14m34.192526518s) [noise=true , address=45.91.116.14:4202 , piece_id=M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.321731813 (526h14m34.192814848s) [noise=true , address=77.90.31.18:4819 , piece_id=YS4DXTHLGCPFOKKLBA2DK2BVOAPTQEGVOPBSRCLFV26SVP6YFHGA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.322221776 (526h14m34.193304811s) [noise=true , address=77.90.31.15:4708 , piece_id=3NX6E54XMZXQAQQE27LTWUSEF5GAIF2XOPQFFAP7LOA2PL7KDL3Q, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.322547776 (526h14m34.193630811s) [noise=true , address=145.79.227.40:4079 , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.322908093 (526h14m34.193991138s) [noise=true , address=145.79.227.29:4975 , piece_id=DAMNBVP7DRSW5OAGYRRP4K5SNJ2WEUT7IAXO77KWRHLQEMBNERZQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.323267968 (526h14m34.194351003s) [noise=true , address=145.79.227.30:4787 , piece_id=OB2OTY3ZEHIUFYM6FY7HIS7HQ353Y2HL6XU4476HXOD26QVA4DPQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.323618165 (526h14m34.19470121s) [noise=true , address=64.49.13.169:4710 , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.324084893 (526h14m34.195167928s) [noise=true , address=193.149.164.143:4694 , piece_id=ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:35.783540136 (526h14m35.654623181s) [upload=461595]: Next returned nil segment Dec 27 13:43:35.783555856 (526h14m35.654638891s) [upload=461595]: Waiting for error group managing segments... Dec 27 13:43:36.192287250 (526h14m36.063370296s) [noise=true , address=145.79.227.29:4952 , piece_id=VXZLF4OGRIR3MYQMMLKLTD235A55F4YTHB3D4YCVCQUL4FZTSFJQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.200200289 (526h14m36.071283334s) [noise=true , address=145.79.227.46:4621 , piece_id=XZ4MKAZAGQKVEJJGSZ5QGT4AT3MFXXNMXICVFCI6SP3E4EVEHZGQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.238087532 (526h14m36.109170577s) [noise=true , address=145.79.227.29:4975 , piece_id=DAMNBVP7DRSW5OAGYRRP4K5SNJ2WEUT7IAXO77KWRHLQEMBNERZQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.379631346 (526h14m36.250714391s) [noise=true , address=77.90.31.19:4869 , piece_id=HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.382945286 (526h14m36.254028321s) [noise=true , address=145.79.227.34:4909 , piece_id=Z6RF5PSOGOZMOXPQYXHMXGFXIELVXXVU4NXXRTAQCIELQXLMVXBQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.400290103 (526h14m36.271373138s) [noise=true , address=77.90.31.18:4819 , piece_id=YS4DXTHLGCPFOKKLBA2DK2BVOAPTQEGVOPBSRCLFV26SVP6YFHGA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.436707208 (526h14m36.307790243s) [noise=true , address=45.91.116.14:4202 , piece_id=M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.487651475 (526h14m36.35873451s) [noise=true , address=45.91.116.12:4234 , piece_id=Y2FIVYLZBQAYBO2TDGIZ4RDQHQJ3WOQH565A7ESIKANQQTQDFQZQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.500850769 (526h14m36.371933814s) [noise=true , address=77.90.31.11:4569 , piece_id=YC2Q3HPJDVY3NEDZNC2Z4KVN6S6DDDGRARASF7KALFAOQ335A7HA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.518828551 (526h14m36.389911586s) [noise=true , address=77.90.31.15:4708 , piece_id=3NX6E54XMZXQAQQE27LTWUSEF5GAIF2XOPQFFAP7LOA2PL7KDL3Q, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.051607243 (526h14m36.922690289s) [noise=true , address=145.79.227.44:4228 , piece_id=6SAYEAOTTCOUFPU75C7YRIULWJGXL4PK2YKCKFB47EUBWMRQQQOQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.096826909 (526h14m36.967909954s) [noise=true , address=145.79.227.30:4787 , piece_id=OB2OTY3ZEHIUFYM6FY7HIS7HQ353Y2HL6XU4476HXOD26QVA4DPQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.140738995 (526h14m37.01182203s) [noise=true , address=77.90.31.21:4899 , piece_id=SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.323613990 (526h14m37.194697035s) [noise=true , address=193.149.164.143:4694 , piece_id=ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.323649618 (526h14m37.194732663s) [seg_pos={0 0}, upload=461595]: Segment reached optimal threshold of 14 pieces. Dec 27 13:43:37.323852865 (526h14m37.1949359s) [noise=true , address=45.91.116.10:4113 , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12puRMny4nbgv2by4tAbXcBBSgG2d6BfwcDDRDcFzEdjSVUJxtd): protocol: expected piece hash; context canceled; EOF Dec 27 13:43:37.323860349 (526h14m37.194943384s) [noise=true , address=45.91.116.10:4113 , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324005135 (526h14m37.1950882s) [noise=true , address=145.79.227.40:4079 , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RYKzn5zVSmxVbWwnBdX8oqtG1WE2MhPeEiWLf5dZeFAut6s2Y): protocol: expected piece hash; context canceled; EOF Dec 27 13:43:37.324022659 (526h14m37.195105704s) [noise=true , address=145.79.227.40:4079 , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324143378 (526h14m37.195226413s) [noise=true , address=77.90.31.16:4739 , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129B3N7yjJvwCzoVLTqWsPbcLQ5TQUyAzg1RCkbSAVKPWYTG8eb): protocol: expected piece hash; context canceled; EOF Dec 27 13:43:37.324154520 (526h14m37.195237555s) [noise=true , address=77.90.31.16:4739 , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324213442 (526h14m37.195296487s) [noise=true , address=64.49.13.169:4710 , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection Dec 27 13:43:37.324249360 (526h14m37.195332405s) [noise=true , address=64.49.13.169:4710 , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324275360 (526h14m37.195358405s) [seg_pos={0 0}, upload=461595]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 13:43:37.324374018 (526h14m37.195457053s) [upload=461595]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:43:37.324381291 (526h14m37.195464337s) [upload=461595]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:43:37.679036406 (526h14m37.550119451s) [upload=461595]: Done uploading segments... Dec 27 13:43:37.679056985 (526h14m37.55014003s) [upload=461595]: Upload finished. err: <nil> Duration: 52.13917431s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 37 out of 60 is failed (threshold=6) [.XXXXX..XXXXX....XX....X.X.X.X.X..XXX.XX.XX.X.XX.X.XXXXXXXXX] duration: 5.001061291s trace: https://jaeger.dev.storj.io/trace/1a54afb4d2ff04bb FAIL (2025-12-27 13:58:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001082873s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........X...X...X..........X.............XX..............] duration: 35.967608736s trace: https://jaeger.dev.storj.io/trace/3ed1e657d506c4ee FAIL (2025-12-27 13:57:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, Piece ID: HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ) context deadline exceeded; piecestore: (Node ID: 12wXqePK1kK7iBo6wHcbqkRmkED8LGJ3rzQNJAWa14cHYhx1oyB, Piece ID: M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA) context deadline exceeded; piecestore: (Node ID: 12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx, Piece ID: ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ) context deadline exceeded; piecestore: (Node ID: 12UjFkFn2UEvr57LDorcijDCvkBkkKitgZEFx6Yqfk5gMStQvJG, Piece ID: SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ) context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.16:4739 , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.319646322 (526h14m34.190729357s) [noise=true , address=77.90.31.21:4899 , piece_id=SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.319933248 (526h14m34.191016283s) [noise=true , address=45.91.116.12:4234 , piece_id=Y2FIVYLZBQAYBO2TDGIZ4RDQHQJ3WOQH565A7ESIKANQQTQDFQZQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320400588 (526h14m34.191483633s) [noise=true , address=145.79.227.29:4952 , piece_id=VXZLF4OGRIR3MYQMMLKLTD235A55F4YTHB3D4YCVCQUL4FZTSFJQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320407341 (526h14m34.191490386s) [noise=true , address=145.79.227.46:4621 , piece_id=XZ4MKAZAGQKVEJJGSZ5QGT4AT3MFXXNMXICVFCI6SP3E4EVEHZGQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320675030 (526h14m34.191758075s) [noise=true , address=45.91.116.10:4113 , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320802022 (526h14m34.191885067s) [noise=true , address=77.90.31.11:4569 , piece_id=YC2Q3HPJDVY3NEDZNC2Z4KVN6S6DDDGRARASF7KALFAOQ335A7HA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.321169643 (526h14m34.192252678s) [noise=true , address=77.90.31.19:4869 , piece_id=HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.321443483 (526h14m34.192526518s) [noise=true , address=45.91.116.14:4202 , piece_id=M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.321731813 (526h14m34.192814848s) [noise=true , address=77.90.31.18:4819 , piece_id=YS4DXTHLGCPFOKKLBA2DK2BVOAPTQEGVOPBSRCLFV26SVP6YFHGA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.322221776 (526h14m34.193304811s) [noise=true , address=77.90.31.15:4708 , piece_id=3NX6E54XMZXQAQQE27LTWUSEF5GAIF2XOPQFFAP7LOA2PL7KDL3Q, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.322547776 (526h14m34.193630811s) [noise=true , address=145.79.227.40:4079 , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.322908093 (526h14m34.193991138s) [noise=true , address=145.79.227.29:4975 , piece_id=DAMNBVP7DRSW5OAGYRRP4K5SNJ2WEUT7IAXO77KWRHLQEMBNERZQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.323267968 (526h14m34.194351003s) [noise=true , address=145.79.227.30:4787 , piece_id=OB2OTY3ZEHIUFYM6FY7HIS7HQ353Y2HL6XU4476HXOD26QVA4DPQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.323618165 (526h14m34.19470121s) [noise=true , address=64.49.13.169:4710 , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.324084893 (526h14m34.195167928s) [noise=true , address=193.149.164.143:4694 , piece_id=ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:35.783540136 (526h14m35.654623181s) [upload=461595]: Next returned nil segment Dec 27 13:43:35.783555856 (526h14m35.654638891s) [upload=461595]: Waiting for error group managing segments... Dec 27 13:43:36.192287250 (526h14m36.063370296s) [noise=true , address=145.79.227.29:4952 , piece_id=VXZLF4OGRIR3MYQMMLKLTD235A55F4YTHB3D4YCVCQUL4FZTSFJQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.200200289 (526h14m36.071283334s) [noise=true , address=145.79.227.46:4621 , piece_id=XZ4MKAZAGQKVEJJGSZ5QGT4AT3MFXXNMXICVFCI6SP3E4EVEHZGQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.238087532 (526h14m36.109170577s) [noise=true , address=145.79.227.29:4975 , piece_id=DAMNBVP7DRSW5OAGYRRP4K5SNJ2WEUT7IAXO77KWRHLQEMBNERZQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.379631346 (526h14m36.250714391s) [noise=true , address=77.90.31.19:4869 , piece_id=HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.382945286 (526h14m36.254028321s) [noise=true , address=145.79.227.34:4909 , piece_id=Z6RF5PSOGOZMOXPQYXHMXGFXIELVXXVU4NXXRTAQCIELQXLMVXBQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.400290103 (526h14m36.271373138s) [noise=true , address=77.90.31.18:4819 , piece_id=YS4DXTHLGCPFOKKLBA2DK2BVOAPTQEGVOPBSRCLFV26SVP6YFHGA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.436707208 (526h14m36.307790243s) [noise=true , address=45.91.116.14:4202 , piece_id=M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.487651475 (526h14m36.35873451s) [noise=true , address=45.91.116.12:4234 , piece_id=Y2FIVYLZBQAYBO2TDGIZ4RDQHQJ3WOQH565A7ESIKANQQTQDFQZQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.500850769 (526h14m36.371933814s) [noise=true , address=77.90.31.11:4569 , piece_id=YC2Q3HPJDVY3NEDZNC2Z4KVN6S6DDDGRARASF7KALFAOQ335A7HA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.518828551 (526h14m36.389911586s) [noise=true , address=77.90.31.15:4708 , piece_id=3NX6E54XMZXQAQQE27LTWUSEF5GAIF2XOPQFFAP7LOA2PL7KDL3Q, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.051607243 (526h14m36.922690289s) [noise=true , address=145.79.227.44:4228 , piece_id=6SAYEAOTTCOUFPU75C7YRIULWJGXL4PK2YKCKFB47EUBWMRQQQOQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.096826909 (526h14m36.967909954s) [noise=true , address=145.79.227.30:4787 , piece_id=OB2OTY3ZEHIUFYM6FY7HIS7HQ353Y2HL6XU4476HXOD26QVA4DPQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.140738995 (526h14m37.01182203s) [noise=true , address=77.90.31.21:4899 , piece_id=SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.323613990 (526h14m37.194697035s) [noise=true , address=193.149.164.143:4694 , piece_id=ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.323649618 (526h14m37.194732663s) [seg_pos={0 0}, upload=461595]: Segment reached optimal threshold of 14 pieces. Dec 27 13:43:37.323852865 (526h14m37.1949359s) [noise=true , address=45.91.116.10:4113 , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12puRMny4nbgv2by4tAbXcBBSgG2d6BfwcDDRDcFzEdjSVUJxtd): protocol: expected piece hash; context canceled; EOF Dec 27 13:43:37.323860349 (526h14m37.194943384s) [noise=true , address=45.91.116.10:4113 , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324005135 (526h14m37.1950882s) [noise=true , address=145.79.227.40:4079 , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RYKzn5zVSmxVbWwnBdX8oqtG1WE2MhPeEiWLf5dZeFAut6s2Y): protocol: expected piece hash; context canceled; EOF Dec 27 13:43:37.324022659 (526h14m37.195105704s) [noise=true , address=145.79.227.40:4079 , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324143378 (526h14m37.195226413s) [noise=true , address=77.90.31.16:4739 , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129B3N7yjJvwCzoVLTqWsPbcLQ5TQUyAzg1RCkbSAVKPWYTG8eb): protocol: expected piece hash; context canceled; EOF Dec 27 13:43:37.324154520 (526h14m37.195237555s) [noise=true , address=77.90.31.16:4739 , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324213442 (526h14m37.195296487s) [noise=true , address=64.49.13.169:4710 , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection Dec 27 13:43:37.324249360 (526h14m37.195332405s) [noise=true , address=64.49.13.169:4710 , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324275360 (526h14m37.195358405s) [seg_pos={0 0}, upload=461595]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 13:43:37.324374018 (526h14m37.195457053s) [upload=461595]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:43:37.324381291 (526h14m37.195464337s) [upload=461595]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:43:37.679036406 (526h14m37.550119451s) [upload=461595]: Done uploading segments... Dec 27 13:43:37.679056985 (526h14m37.55014003s) [upload=461595]: Upload finished. err: <nil> Duration: 35.967632313s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.002081751s trace: https://jaeger.dev.storj.io/trace/3079997b6e8f85b7 FAIL (2025-12-27 13:57:42 UTC) orbiter warning: context deadline exceeded Duration: 30.002107219s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X...X.XXX....X...X...X...XX.......X....X..XX.X.XXXX..X..X...] duration: 5.920681124s trace: https://jaeger.dev.storj.io/trace/61cbbcf7e1004bc3 FAIL (2025-12-27 13:57:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: WRCB4QABBNPHWTPXEEO4SLB7MR3MT467XFOZTAWCCWG67ZAAK4GA) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: 2X3DL3V773MFFXYVQYRIYKXIRU4JZI4KELFIXZS3PUTBPZVTGKRA) context deadline exceeded ======LOGS====== ing segments... Dec 27 13:55:39.506895724 (526h26m39.377978769s) [upload=461769]: Got next segment. Inline: false Dec 27 13:55:39.506937954 (526h26m39.378021s) [upload=461769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:55:39.506943695 (526h26m39.37802673s) [upload=461769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:55:39.746845108 (526h26m39.617928254s) [seg_pos={0 0}, upload=461769]: Begin upload segment... Dec 27 13:55:39.746972181 (526h26m39.618055226s) [seg_pos={0 0}, upload=461769]: Done begin upload segment. Dec 27 13:55:39.747039980 (526h26m39.618123025s) [noise=true , address=145.79.227.23:4546 , piece_id=SFSPRBAS5DZ7PGSPKN3F5RV4I2WPP6PXBQSZTIEVF2OZQIBY7PIQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.747055749 (526h26m39.618138794s) [noise=true , address=145.79.227.16:4912 , piece_id=2V6SNFMGJJF6PX7YA63PQUHCWEBEQQRXTIV5YP5IXPGOHL57DOWQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.747668517 (526h26m39.618751562s) [noise=true , address=145.79.227.28:4727 , piece_id=NSE2QHV2T375DF7JQO32WROCMIA3HUVBNBZZW4LCBCJ2K4CAL23Q, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.748154081 (526h26m39.619237136s) [noise=true , address=45.91.116.12:4225 , piece_id=NMARGNK6INEKMWLK2IQSTSMAVSE36E3T4WXMD6DEQMJKYTB6ZKXA, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.748539615 (526h26m39.61962266s) [noise=true , address=145.79.227.41:4085 , piece_id=4T7J4A56XHXQ7J5MP3PB5GUPL6ZQRYLTZ66IMD3GJVE4EGQY5JIA, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.748812355 (526h26m39.6198954s) [noise=true , address=77.90.31.12:4589 , piece_id=WRCB4QABBNPHWTPXEEO4SLB7MR3MT467XFOZTAWCCWG67ZAAK4GA, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.749151961 (526h26m39.620235036s) [noise=true , address=145.79.227.47:4267 , piece_id=TUZIXXBICDEZ64AI7XOVBBQRKM4WS7UVXOSFLGUXRZZXZJ5QFRZQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.749662393 (526h26m39.620745438s) [noise=true , address=145.79.227.33:4869 , piece_id=MCLA62SXY6NLKAE5AJJQS4EKPY6NH6CO4HNSEW4HU5PMAALAEDUA, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.750118381 (526h26m39.621201416s) [noise=true , address=145.79.227.27:4689 , piece_id=KCW5QW3O36SBVXA6BJV5DIRTS4HOYMQSAFRXCCCNHB3OPAXXFCFQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.750451205 (526h26m39.62153424s) [noise=true , address=145.79.227.37:4567 , piece_id=REVK53YXB7BT7T2IXTHM4NQBLC5LD2V6Q3AIX5I4O5Y2V5VQJDHQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.750760544 (526h26m39.62184358s) [noise=true , address=45.91.116.13:4153 , piece_id=VSAPME7HWTPXGRIHNZOOW7CHLHC55ELZUVRHNN7A7SHEVEEJEOYQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.751022002 (526h26m39.622105077s) [noise=true , address=145.79.227.29:4942 , piece_id=XQY6IDGGUO73AYWSQPJ5B2K724RWGVNDNRKPUBUKOGR7WPLBLWQQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.751446380 (526h26m39.622529415s) [noise=true , address=145.79.227.13:4804 , piece_id=LSHRLHBDQ5XYJCZ4XCOKY47EWU34SRRJWEORJS4PF4BTU7HA6DAA, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.751778563 (526h26m39.622861608s) [noise=true , address=77.90.31.11:4556 , piece_id=2X3DL3V773MFFXYVQYRIYKXIRU4JZI4KELFIXZS3PUTBPZVTGKRA, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.752151403 (526h26m39.623234438s) [noise=true , address=145.79.227.45:4231 , piece_id=ZI2FWBYUL4G7PCN5YZWAELM22AQP2O7Z6HML5NMGK5UJ3RSTEODQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.752460461 (526h26m39.623543496s) [noise=true , address=77.90.31.19:4842 , piece_id=OGWSBYPGFHPWZ5VJISQ7A3PSYN7GGS6QVYINEAWBGAP4ADKVMPAA, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:41.454144697 (526h26m41.325227742s) [upload=461769]: Next returned nil segment Dec 27 13:55:41.454167932 (526h26m41.325250977s) [upload=461769]: Waiting for error group managing segments... Dec 27 13:55:41.738963230 (526h26m41.610046265s) [noise=true , address=145.79.227.37:4567 , piece_id=REVK53YXB7BT7T2IXTHM4NQBLC5LD2V6Q3AIX5I4O5Y2V5VQJDHQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.742975230 (526h26m41.614058265s) [noise=true , address=145.79.227.27:4689 , piece_id=KCW5QW3O36SBVXA6BJV5DIRTS4HOYMQSAFRXCCCNHB3OPAXXFCFQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.751149826 (526h26m41.622232871s) [noise=true , address=145.79.227.41:4085 , piece_id=4T7J4A56XHXQ7J5MP3PB5GUPL6ZQRYLTZ66IMD3GJVE4EGQY5JIA, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.751779916 (526h26m41.622862951s) [noise=true , address=145.79.227.45:4231 , piece_id=ZI2FWBYUL4G7PCN5YZWAELM22AQP2O7Z6HML5NMGK5UJ3RSTEODQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.755286142 (526h26m41.626369177s) [noise=true , address=145.79.227.16:4912 , piece_id=2V6SNFMGJJF6PX7YA63PQUHCWEBEQQRXTIV5YP5IXPGOHL57DOWQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.763271658 (526h26m41.634354693s) [noise=true , address=145.79.227.13:4804 , piece_id=LSHRLHBDQ5XYJCZ4XCOKY47EWU34SRRJWEORJS4PF4BTU7HA6DAA, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.764016025 (526h26m41.63509906s) [noise=true , address=145.79.227.28:4727 , piece_id=NSE2QHV2T375DF7JQO32WROCMIA3HUVBNBZZW4LCBCJ2K4CAL23Q, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.764095897 (526h26m41.635178932s) [noise=true , address=145.79.227.29:4942 , piece_id=XQY6IDGGUO73AYWSQPJ5B2K724RWGVNDNRKPUBUKOGR7WPLBLWQQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.778988115 (526h26m41.65007116s) [noise=true , address=145.79.227.47:4267 , piece_id=TUZIXXBICDEZ64AI7XOVBBQRKM4WS7UVXOSFLGUXRZZXZJ5QFRZQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.783665982 (526h26m41.654749027s) [noise=true , address=145.79.227.23:4546 , piece_id=SFSPRBAS5DZ7PGSPKN3F5RV4I2WPP6PXBQSZTIEVF2OZQIBY7PIQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:42.700160844 (526h26m42.571243879s) [noise=true , address=77.90.31.11:4556 , piece_id=2X3DL3V773MFFXYVQYRIYKXIRU4JZI4KELFIXZS3PUTBPZVTGKRA, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:42.928495633 (526h26m42.799578678s) [noise=true , address=77.90.31.19:4842 , piece_id=OGWSBYPGFHPWZ5VJISQ7A3PSYN7GGS6QVYINEAWBGAP4ADKVMPAA, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:43.041556590 (526h26m42.912639625s) [noise=true , address=77.90.31.12:4589 , piece_id=WRCB4QABBNPHWTPXEEO4SLB7MR3MT467XFOZTAWCCWG67ZAAK4GA, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:43.041576267 (526h26m42.912659302s) [seg_pos={0 0}, upload=461769]: Segment reached optimal threshold of 13 pieces. Dec 27 13:55:43.041779755 (526h26m42.91286279s) [noise=true , address=145.79.227.33:4869 , piece_id=MCLA62SXY6NLKAE5AJJQS4EKPY6NH6CO4HNSEW4HU5PMAALAEDUA, seg_pos={0 0}, upload=461769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 27 13:55:43.041785526 (526h26m42.912868561s) [noise=true , address=145.79.227.33:4869 , piece_id=MCLA62SXY6NLKAE5AJJQS4EKPY6NH6CO4HNSEW4HU5PMAALAEDUA, seg_pos={0 0}, upload=461769]: Long tail cancellation due to optimal threshold Dec 27 13:55:43.042105304 (526h26m42.91318837s) [noise=true , address=45.91.116.13:4153 , piece_id=VSAPME7HWTPXGRIHNZOOW7CHLHC55ELZUVRHNN7A7SHEVEEJEOYQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:36608->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:36608->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36608->45.91.116.13:4153: use of closed network connection Dec 27 13:55:43.042131284 (526h26m42.913214319s) [noise=true , address=45.91.116.13:4153 , piece_id=VSAPME7HWTPXGRIHNZOOW7CHLHC55ELZUVRHNN7A7SHEVEEJEOYQ, seg_pos={0 0}, upload=461769]: Long tail cancellation due to optimal threshold Dec 27 13:55:43.042255090 (526h26m42.913338125s) [noise=true , address=45.91.116.12:4225 , piece_id=NMARGNK6INEKMWLK2IQSTSMAVSE36E3T4WXMD6DEQMJKYTB6ZKXA, seg_pos={0 0}, upload=461769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:47484->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:47484->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47484->45.91.116.12:4225: use of closed network connection Dec 27 13:55:43.042264498 (526h26m42.913347533s) [noise=true , address=45.91.116.12:4225 , piece_id=NMARGNK6INEKMWLK2IQSTSMAVSE36E3T4WXMD6DEQMJKYTB6ZKXA, seg_pos={0 0}, upload=461769]: Long tail cancellation due to optimal threshold Dec 27 13:55:43.042283343 (526h26m42.913366388s) [seg_pos={0 0}, upload=461769]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:55:43.042387552 (526h26m42.913470597s) [upload=461769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:55:43.042394376 (526h26m42.913477411s) [upload=461769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:55:43.323639824 (526h26m43.194722859s) [upload=461769]: Done uploading segments... Dec 27 13:55:43.323658681 (526h26m43.194741716s) [upload=461769]: Upload finished. err: <nil> Duration: 5.920713376s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [..XXXXX..XXXXX....XX....X.X.X.X.X..XXX.XX.XX.X.XX.X.XXXXXXXX] duration: 5.0010074s trace: https://jaeger.dev.storj.io/trace/3a027cf802c4fb43 FAIL (2025-12-27 13:57:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001030504s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [............X...X...X..........X.............XX.............] duration: 43.503586725s trace: https://jaeger.dev.storj.io/trace/3ed1e657d506c4ee FAIL (2025-12-27 13:56:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, Piece ID: HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ) context deadline exceeded; piecestore: (Node ID: 12wXqePK1kK7iBo6wHcbqkRmkED8LGJ3rzQNJAWa14cHYhx1oyB, Piece ID: M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA) context deadline exceeded; piecestore: (Node ID: 12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx, Piece ID: ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ) context deadline exceeded; piecestore: (Node ID: 12UjFkFn2UEvr57LDorcijDCvkBkkKitgZEFx6Yqfk5gMStQvJG, Piece ID: SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ) context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.16:4739 , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.319646322 (526h14m34.190729357s) [noise=true , address=77.90.31.21:4899 , piece_id=SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.319933248 (526h14m34.191016283s) [noise=true , address=45.91.116.12:4234 , piece_id=Y2FIVYLZBQAYBO2TDGIZ4RDQHQJ3WOQH565A7ESIKANQQTQDFQZQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320400588 (526h14m34.191483633s) [noise=true , address=145.79.227.29:4952 , piece_id=VXZLF4OGRIR3MYQMMLKLTD235A55F4YTHB3D4YCVCQUL4FZTSFJQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320407341 (526h14m34.191490386s) [noise=true , address=145.79.227.46:4621 , piece_id=XZ4MKAZAGQKVEJJGSZ5QGT4AT3MFXXNMXICVFCI6SP3E4EVEHZGQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320675030 (526h14m34.191758075s) [noise=true , address=45.91.116.10:4113 , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320802022 (526h14m34.191885067s) [noise=true , address=77.90.31.11:4569 , piece_id=YC2Q3HPJDVY3NEDZNC2Z4KVN6S6DDDGRARASF7KALFAOQ335A7HA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.321169643 (526h14m34.192252678s) [noise=true , address=77.90.31.19:4869 , piece_id=HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.321443483 (526h14m34.192526518s) [noise=true , address=45.91.116.14:4202 , piece_id=M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.321731813 (526h14m34.192814848s) [noise=true , address=77.90.31.18:4819 , piece_id=YS4DXTHLGCPFOKKLBA2DK2BVOAPTQEGVOPBSRCLFV26SVP6YFHGA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.322221776 (526h14m34.193304811s) [noise=true , address=77.90.31.15:4708 , piece_id=3NX6E54XMZXQAQQE27LTWUSEF5GAIF2XOPQFFAP7LOA2PL7KDL3Q, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.322547776 (526h14m34.193630811s) [noise=true , address=145.79.227.40:4079 , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.322908093 (526h14m34.193991138s) [noise=true , address=145.79.227.29:4975 , piece_id=DAMNBVP7DRSW5OAGYRRP4K5SNJ2WEUT7IAXO77KWRHLQEMBNERZQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.323267968 (526h14m34.194351003s) [noise=true , address=145.79.227.30:4787 , piece_id=OB2OTY3ZEHIUFYM6FY7HIS7HQ353Y2HL6XU4476HXOD26QVA4DPQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.323618165 (526h14m34.19470121s) [noise=true , address=64.49.13.169:4710 , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.324084893 (526h14m34.195167928s) [noise=true , address=193.149.164.143:4694 , piece_id=ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:35.783540136 (526h14m35.654623181s) [upload=461595]: Next returned nil segment Dec 27 13:43:35.783555856 (526h14m35.654638891s) [upload=461595]: Waiting for error group managing segments... Dec 27 13:43:36.192287250 (526h14m36.063370296s) [noise=true , address=145.79.227.29:4952 , piece_id=VXZLF4OGRIR3MYQMMLKLTD235A55F4YTHB3D4YCVCQUL4FZTSFJQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.200200289 (526h14m36.071283334s) [noise=true , address=145.79.227.46:4621 , piece_id=XZ4MKAZAGQKVEJJGSZ5QGT4AT3MFXXNMXICVFCI6SP3E4EVEHZGQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.238087532 (526h14m36.109170577s) [noise=true , address=145.79.227.29:4975 , piece_id=DAMNBVP7DRSW5OAGYRRP4K5SNJ2WEUT7IAXO77KWRHLQEMBNERZQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.379631346 (526h14m36.250714391s) [noise=true , address=77.90.31.19:4869 , piece_id=HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.382945286 (526h14m36.254028321s) [noise=true , address=145.79.227.34:4909 , piece_id=Z6RF5PSOGOZMOXPQYXHMXGFXIELVXXVU4NXXRTAQCIELQXLMVXBQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.400290103 (526h14m36.271373138s) [noise=true , address=77.90.31.18:4819 , piece_id=YS4DXTHLGCPFOKKLBA2DK2BVOAPTQEGVOPBSRCLFV26SVP6YFHGA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.436707208 (526h14m36.307790243s) [noise=true , address=45.91.116.14:4202 , piece_id=M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.487651475 (526h14m36.35873451s) [noise=true , address=45.91.116.12:4234 , piece_id=Y2FIVYLZBQAYBO2TDGIZ4RDQHQJ3WOQH565A7ESIKANQQTQDFQZQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.500850769 (526h14m36.371933814s) [noise=true , address=77.90.31.11:4569 , piece_id=YC2Q3HPJDVY3NEDZNC2Z4KVN6S6DDDGRARASF7KALFAOQ335A7HA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.518828551 (526h14m36.389911586s) [noise=true , address=77.90.31.15:4708 , piece_id=3NX6E54XMZXQAQQE27LTWUSEF5GAIF2XOPQFFAP7LOA2PL7KDL3Q, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.051607243 (526h14m36.922690289s) [noise=true , address=145.79.227.44:4228 , piece_id=6SAYEAOTTCOUFPU75C7YRIULWJGXL4PK2YKCKFB47EUBWMRQQQOQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.096826909 (526h14m36.967909954s) [noise=true , address=145.79.227.30:4787 , piece_id=OB2OTY3ZEHIUFYM6FY7HIS7HQ353Y2HL6XU4476HXOD26QVA4DPQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.140738995 (526h14m37.01182203s) [noise=true , address=77.90.31.21:4899 , piece_id=SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.323613990 (526h14m37.194697035s) [noise=true , address=193.149.164.143:4694 , piece_id=ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.323649618 (526h14m37.194732663s) [seg_pos={0 0}, upload=461595]: Segment reached optimal threshold of 14 pieces. Dec 27 13:43:37.323852865 (526h14m37.1949359s) [noise=true , address=45.91.116.10:4113 , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12puRMny4nbgv2by4tAbXcBBSgG2d6BfwcDDRDcFzEdjSVUJxtd): protocol: expected piece hash; context canceled; EOF Dec 27 13:43:37.323860349 (526h14m37.194943384s) [noise=true , address=45.91.116.10:4113 , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324005135 (526h14m37.1950882s) [noise=true , address=145.79.227.40:4079 , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RYKzn5zVSmxVbWwnBdX8oqtG1WE2MhPeEiWLf5dZeFAut6s2Y): protocol: expected piece hash; context canceled; EOF Dec 27 13:43:37.324022659 (526h14m37.195105704s) [noise=true , address=145.79.227.40:4079 , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324143378 (526h14m37.195226413s) [noise=true , address=77.90.31.16:4739 , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129B3N7yjJvwCzoVLTqWsPbcLQ5TQUyAzg1RCkbSAVKPWYTG8eb): protocol: expected piece hash; context canceled; EOF Dec 27 13:43:37.324154520 (526h14m37.195237555s) [noise=true , address=77.90.31.16:4739 , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324213442 (526h14m37.195296487s) [noise=true , address=64.49.13.169:4710 , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection Dec 27 13:43:37.324249360 (526h14m37.195332405s) [noise=true , address=64.49.13.169:4710 , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324275360 (526h14m37.195358405s) [seg_pos={0 0}, upload=461595]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 13:43:37.324374018 (526h14m37.195457053s) [upload=461595]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:43:37.324381291 (526h14m37.195464337s) [upload=461595]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:43:37.679036406 (526h14m37.550119451s) [upload=461595]: Done uploading segments... Dec 27 13:43:37.679056985 (526h14m37.55014003s) [upload=461595]: Upload finished. err: <nil> Duration: 43.503624145s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [...XXXXX..XXXXX....XX....X.X.X.X.X..XXX.XX.XX.X.XX.X.XXXXXXX] duration: 5.001542918s trace: https://jaeger.dev.storj.io/trace/14aee764eb3a5c91 FAIL (2025-12-27 13:56:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001563749s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X...X.XXX....X...X...X...XX.......X....X..XX.X.XXXX..X..X..] duration: 8.195142114s trace: https://jaeger.dev.storj.io/trace/61cbbcf7e1004bc3 FAIL (2025-12-27 13:56:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: WRCB4QABBNPHWTPXEEO4SLB7MR3MT467XFOZTAWCCWG67ZAAK4GA) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: 2X3DL3V773MFFXYVQYRIYKXIRU4JZI4KELFIXZS3PUTBPZVTGKRA) context deadline exceeded ======LOGS====== ing segments... Dec 27 13:55:39.506895724 (526h26m39.377978769s) [upload=461769]: Got next segment. Inline: false Dec 27 13:55:39.506937954 (526h26m39.378021s) [upload=461769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:55:39.506943695 (526h26m39.37802673s) [upload=461769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:55:39.746845108 (526h26m39.617928254s) [seg_pos={0 0}, upload=461769]: Begin upload segment... Dec 27 13:55:39.746972181 (526h26m39.618055226s) [seg_pos={0 0}, upload=461769]: Done begin upload segment. Dec 27 13:55:39.747039980 (526h26m39.618123025s) [noise=true , address=145.79.227.23:4546 , piece_id=SFSPRBAS5DZ7PGSPKN3F5RV4I2WPP6PXBQSZTIEVF2OZQIBY7PIQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.747055749 (526h26m39.618138794s) [noise=true , address=145.79.227.16:4912 , piece_id=2V6SNFMGJJF6PX7YA63PQUHCWEBEQQRXTIV5YP5IXPGOHL57DOWQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.747668517 (526h26m39.618751562s) [noise=true , address=145.79.227.28:4727 , piece_id=NSE2QHV2T375DF7JQO32WROCMIA3HUVBNBZZW4LCBCJ2K4CAL23Q, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.748154081 (526h26m39.619237136s) [noise=true , address=45.91.116.12:4225 , piece_id=NMARGNK6INEKMWLK2IQSTSMAVSE36E3T4WXMD6DEQMJKYTB6ZKXA, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.748539615 (526h26m39.61962266s) [noise=true , address=145.79.227.41:4085 , piece_id=4T7J4A56XHXQ7J5MP3PB5GUPL6ZQRYLTZ66IMD3GJVE4EGQY5JIA, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.748812355 (526h26m39.6198954s) [noise=true , address=77.90.31.12:4589 , piece_id=WRCB4QABBNPHWTPXEEO4SLB7MR3MT467XFOZTAWCCWG67ZAAK4GA, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.749151961 (526h26m39.620235036s) [noise=true , address=145.79.227.47:4267 , piece_id=TUZIXXBICDEZ64AI7XOVBBQRKM4WS7UVXOSFLGUXRZZXZJ5QFRZQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.749662393 (526h26m39.620745438s) [noise=true , address=145.79.227.33:4869 , piece_id=MCLA62SXY6NLKAE5AJJQS4EKPY6NH6CO4HNSEW4HU5PMAALAEDUA, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.750118381 (526h26m39.621201416s) [noise=true , address=145.79.227.27:4689 , piece_id=KCW5QW3O36SBVXA6BJV5DIRTS4HOYMQSAFRXCCCNHB3OPAXXFCFQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.750451205 (526h26m39.62153424s) [noise=true , address=145.79.227.37:4567 , piece_id=REVK53YXB7BT7T2IXTHM4NQBLC5LD2V6Q3AIX5I4O5Y2V5VQJDHQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.750760544 (526h26m39.62184358s) [noise=true , address=45.91.116.13:4153 , piece_id=VSAPME7HWTPXGRIHNZOOW7CHLHC55ELZUVRHNN7A7SHEVEEJEOYQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.751022002 (526h26m39.622105077s) [noise=true , address=145.79.227.29:4942 , piece_id=XQY6IDGGUO73AYWSQPJ5B2K724RWGVNDNRKPUBUKOGR7WPLBLWQQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.751446380 (526h26m39.622529415s) [noise=true , address=145.79.227.13:4804 , piece_id=LSHRLHBDQ5XYJCZ4XCOKY47EWU34SRRJWEORJS4PF4BTU7HA6DAA, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.751778563 (526h26m39.622861608s) [noise=true , address=77.90.31.11:4556 , piece_id=2X3DL3V773MFFXYVQYRIYKXIRU4JZI4KELFIXZS3PUTBPZVTGKRA, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.752151403 (526h26m39.623234438s) [noise=true , address=145.79.227.45:4231 , piece_id=ZI2FWBYUL4G7PCN5YZWAELM22AQP2O7Z6HML5NMGK5UJ3RSTEODQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.752460461 (526h26m39.623543496s) [noise=true , address=77.90.31.19:4842 , piece_id=OGWSBYPGFHPWZ5VJISQ7A3PSYN7GGS6QVYINEAWBGAP4ADKVMPAA, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:41.454144697 (526h26m41.325227742s) [upload=461769]: Next returned nil segment Dec 27 13:55:41.454167932 (526h26m41.325250977s) [upload=461769]: Waiting for error group managing segments... Dec 27 13:55:41.738963230 (526h26m41.610046265s) [noise=true , address=145.79.227.37:4567 , piece_id=REVK53YXB7BT7T2IXTHM4NQBLC5LD2V6Q3AIX5I4O5Y2V5VQJDHQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.742975230 (526h26m41.614058265s) [noise=true , address=145.79.227.27:4689 , piece_id=KCW5QW3O36SBVXA6BJV5DIRTS4HOYMQSAFRXCCCNHB3OPAXXFCFQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.751149826 (526h26m41.622232871s) [noise=true , address=145.79.227.41:4085 , piece_id=4T7J4A56XHXQ7J5MP3PB5GUPL6ZQRYLTZ66IMD3GJVE4EGQY5JIA, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.751779916 (526h26m41.622862951s) [noise=true , address=145.79.227.45:4231 , piece_id=ZI2FWBYUL4G7PCN5YZWAELM22AQP2O7Z6HML5NMGK5UJ3RSTEODQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.755286142 (526h26m41.626369177s) [noise=true , address=145.79.227.16:4912 , piece_id=2V6SNFMGJJF6PX7YA63PQUHCWEBEQQRXTIV5YP5IXPGOHL57DOWQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.763271658 (526h26m41.634354693s) [noise=true , address=145.79.227.13:4804 , piece_id=LSHRLHBDQ5XYJCZ4XCOKY47EWU34SRRJWEORJS4PF4BTU7HA6DAA, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.764016025 (526h26m41.63509906s) [noise=true , address=145.79.227.28:4727 , piece_id=NSE2QHV2T375DF7JQO32WROCMIA3HUVBNBZZW4LCBCJ2K4CAL23Q, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.764095897 (526h26m41.635178932s) [noise=true , address=145.79.227.29:4942 , piece_id=XQY6IDGGUO73AYWSQPJ5B2K724RWGVNDNRKPUBUKOGR7WPLBLWQQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.778988115 (526h26m41.65007116s) [noise=true , address=145.79.227.47:4267 , piece_id=TUZIXXBICDEZ64AI7XOVBBQRKM4WS7UVXOSFLGUXRZZXZJ5QFRZQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.783665982 (526h26m41.654749027s) [noise=true , address=145.79.227.23:4546 , piece_id=SFSPRBAS5DZ7PGSPKN3F5RV4I2WPP6PXBQSZTIEVF2OZQIBY7PIQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:42.700160844 (526h26m42.571243879s) [noise=true , address=77.90.31.11:4556 , piece_id=2X3DL3V773MFFXYVQYRIYKXIRU4JZI4KELFIXZS3PUTBPZVTGKRA, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:42.928495633 (526h26m42.799578678s) [noise=true , address=77.90.31.19:4842 , piece_id=OGWSBYPGFHPWZ5VJISQ7A3PSYN7GGS6QVYINEAWBGAP4ADKVMPAA, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:43.041556590 (526h26m42.912639625s) [noise=true , address=77.90.31.12:4589 , piece_id=WRCB4QABBNPHWTPXEEO4SLB7MR3MT467XFOZTAWCCWG67ZAAK4GA, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:43.041576267 (526h26m42.912659302s) [seg_pos={0 0}, upload=461769]: Segment reached optimal threshold of 13 pieces. Dec 27 13:55:43.041779755 (526h26m42.91286279s) [noise=true , address=145.79.227.33:4869 , piece_id=MCLA62SXY6NLKAE5AJJQS4EKPY6NH6CO4HNSEW4HU5PMAALAEDUA, seg_pos={0 0}, upload=461769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 27 13:55:43.041785526 (526h26m42.912868561s) [noise=true , address=145.79.227.33:4869 , piece_id=MCLA62SXY6NLKAE5AJJQS4EKPY6NH6CO4HNSEW4HU5PMAALAEDUA, seg_pos={0 0}, upload=461769]: Long tail cancellation due to optimal threshold Dec 27 13:55:43.042105304 (526h26m42.91318837s) [noise=true , address=45.91.116.13:4153 , piece_id=VSAPME7HWTPXGRIHNZOOW7CHLHC55ELZUVRHNN7A7SHEVEEJEOYQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:36608->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:36608->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36608->45.91.116.13:4153: use of closed network connection Dec 27 13:55:43.042131284 (526h26m42.913214319s) [noise=true , address=45.91.116.13:4153 , piece_id=VSAPME7HWTPXGRIHNZOOW7CHLHC55ELZUVRHNN7A7SHEVEEJEOYQ, seg_pos={0 0}, upload=461769]: Long tail cancellation due to optimal threshold Dec 27 13:55:43.042255090 (526h26m42.913338125s) [noise=true , address=45.91.116.12:4225 , piece_id=NMARGNK6INEKMWLK2IQSTSMAVSE36E3T4WXMD6DEQMJKYTB6ZKXA, seg_pos={0 0}, upload=461769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:47484->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:47484->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47484->45.91.116.12:4225: use of closed network connection Dec 27 13:55:43.042264498 (526h26m42.913347533s) [noise=true , address=45.91.116.12:4225 , piece_id=NMARGNK6INEKMWLK2IQSTSMAVSE36E3T4WXMD6DEQMJKYTB6ZKXA, seg_pos={0 0}, upload=461769]: Long tail cancellation due to optimal threshold Dec 27 13:55:43.042283343 (526h26m42.913366388s) [seg_pos={0 0}, upload=461769]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:55:43.042387552 (526h26m42.913470597s) [upload=461769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:55:43.042394376 (526h26m42.913477411s) [upload=461769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:55:43.323639824 (526h26m43.194722859s) [upload=461769]: Done uploading segments... Dec 27 13:55:43.323658681 (526h26m43.194741716s) [upload=461769]: Upload finished. err: <nil> Duration: 8.195165329s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..X...X.XXX....X...X...X...XX.......X....X..XX.X.XXXX..X..X.] duration: 5.892872691s trace: https://jaeger.dev.storj.io/trace/61cbbcf7e1004bc3 FAIL (2025-12-27 13:56:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: WRCB4QABBNPHWTPXEEO4SLB7MR3MT467XFOZTAWCCWG67ZAAK4GA) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: 2X3DL3V773MFFXYVQYRIYKXIRU4JZI4KELFIXZS3PUTBPZVTGKRA) context deadline exceeded ======LOGS====== ing segments... Dec 27 13:55:39.506895724 (526h26m39.377978769s) [upload=461769]: Got next segment. Inline: false Dec 27 13:55:39.506937954 (526h26m39.378021s) [upload=461769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:55:39.506943695 (526h26m39.37802673s) [upload=461769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:55:39.746845108 (526h26m39.617928254s) [seg_pos={0 0}, upload=461769]: Begin upload segment... Dec 27 13:55:39.746972181 (526h26m39.618055226s) [seg_pos={0 0}, upload=461769]: Done begin upload segment. Dec 27 13:55:39.747039980 (526h26m39.618123025s) [noise=true , address=145.79.227.23:4546 , piece_id=SFSPRBAS5DZ7PGSPKN3F5RV4I2WPP6PXBQSZTIEVF2OZQIBY7PIQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.747055749 (526h26m39.618138794s) [noise=true , address=145.79.227.16:4912 , piece_id=2V6SNFMGJJF6PX7YA63PQUHCWEBEQQRXTIV5YP5IXPGOHL57DOWQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.747668517 (526h26m39.618751562s) [noise=true , address=145.79.227.28:4727 , piece_id=NSE2QHV2T375DF7JQO32WROCMIA3HUVBNBZZW4LCBCJ2K4CAL23Q, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.748154081 (526h26m39.619237136s) [noise=true , address=45.91.116.12:4225 , piece_id=NMARGNK6INEKMWLK2IQSTSMAVSE36E3T4WXMD6DEQMJKYTB6ZKXA, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.748539615 (526h26m39.61962266s) [noise=true , address=145.79.227.41:4085 , piece_id=4T7J4A56XHXQ7J5MP3PB5GUPL6ZQRYLTZ66IMD3GJVE4EGQY5JIA, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.748812355 (526h26m39.6198954s) [noise=true , address=77.90.31.12:4589 , piece_id=WRCB4QABBNPHWTPXEEO4SLB7MR3MT467XFOZTAWCCWG67ZAAK4GA, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.749151961 (526h26m39.620235036s) [noise=true , address=145.79.227.47:4267 , piece_id=TUZIXXBICDEZ64AI7XOVBBQRKM4WS7UVXOSFLGUXRZZXZJ5QFRZQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.749662393 (526h26m39.620745438s) [noise=true , address=145.79.227.33:4869 , piece_id=MCLA62SXY6NLKAE5AJJQS4EKPY6NH6CO4HNSEW4HU5PMAALAEDUA, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.750118381 (526h26m39.621201416s) [noise=true , address=145.79.227.27:4689 , piece_id=KCW5QW3O36SBVXA6BJV5DIRTS4HOYMQSAFRXCCCNHB3OPAXXFCFQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.750451205 (526h26m39.62153424s) [noise=true , address=145.79.227.37:4567 , piece_id=REVK53YXB7BT7T2IXTHM4NQBLC5LD2V6Q3AIX5I4O5Y2V5VQJDHQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.750760544 (526h26m39.62184358s) [noise=true , address=45.91.116.13:4153 , piece_id=VSAPME7HWTPXGRIHNZOOW7CHLHC55ELZUVRHNN7A7SHEVEEJEOYQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.751022002 (526h26m39.622105077s) [noise=true , address=145.79.227.29:4942 , piece_id=XQY6IDGGUO73AYWSQPJ5B2K724RWGVNDNRKPUBUKOGR7WPLBLWQQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.751446380 (526h26m39.622529415s) [noise=true , address=145.79.227.13:4804 , piece_id=LSHRLHBDQ5XYJCZ4XCOKY47EWU34SRRJWEORJS4PF4BTU7HA6DAA, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.751778563 (526h26m39.622861608s) [noise=true , address=77.90.31.11:4556 , piece_id=2X3DL3V773MFFXYVQYRIYKXIRU4JZI4KELFIXZS3PUTBPZVTGKRA, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.752151403 (526h26m39.623234438s) [noise=true , address=145.79.227.45:4231 , piece_id=ZI2FWBYUL4G7PCN5YZWAELM22AQP2O7Z6HML5NMGK5UJ3RSTEODQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.752460461 (526h26m39.623543496s) [noise=true , address=77.90.31.19:4842 , piece_id=OGWSBYPGFHPWZ5VJISQ7A3PSYN7GGS6QVYINEAWBGAP4ADKVMPAA, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:41.454144697 (526h26m41.325227742s) [upload=461769]: Next returned nil segment Dec 27 13:55:41.454167932 (526h26m41.325250977s) [upload=461769]: Waiting for error group managing segments... Dec 27 13:55:41.738963230 (526h26m41.610046265s) [noise=true , address=145.79.227.37:4567 , piece_id=REVK53YXB7BT7T2IXTHM4NQBLC5LD2V6Q3AIX5I4O5Y2V5VQJDHQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.742975230 (526h26m41.614058265s) [noise=true , address=145.79.227.27:4689 , piece_id=KCW5QW3O36SBVXA6BJV5DIRTS4HOYMQSAFRXCCCNHB3OPAXXFCFQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.751149826 (526h26m41.622232871s) [noise=true , address=145.79.227.41:4085 , piece_id=4T7J4A56XHXQ7J5MP3PB5GUPL6ZQRYLTZ66IMD3GJVE4EGQY5JIA, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.751779916 (526h26m41.622862951s) [noise=true , address=145.79.227.45:4231 , piece_id=ZI2FWBYUL4G7PCN5YZWAELM22AQP2O7Z6HML5NMGK5UJ3RSTEODQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.755286142 (526h26m41.626369177s) [noise=true , address=145.79.227.16:4912 , piece_id=2V6SNFMGJJF6PX7YA63PQUHCWEBEQQRXTIV5YP5IXPGOHL57DOWQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.763271658 (526h26m41.634354693s) [noise=true , address=145.79.227.13:4804 , piece_id=LSHRLHBDQ5XYJCZ4XCOKY47EWU34SRRJWEORJS4PF4BTU7HA6DAA, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.764016025 (526h26m41.63509906s) [noise=true , address=145.79.227.28:4727 , piece_id=NSE2QHV2T375DF7JQO32WROCMIA3HUVBNBZZW4LCBCJ2K4CAL23Q, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.764095897 (526h26m41.635178932s) [noise=true , address=145.79.227.29:4942 , piece_id=XQY6IDGGUO73AYWSQPJ5B2K724RWGVNDNRKPUBUKOGR7WPLBLWQQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.778988115 (526h26m41.65007116s) [noise=true , address=145.79.227.47:4267 , piece_id=TUZIXXBICDEZ64AI7XOVBBQRKM4WS7UVXOSFLGUXRZZXZJ5QFRZQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.783665982 (526h26m41.654749027s) [noise=true , address=145.79.227.23:4546 , piece_id=SFSPRBAS5DZ7PGSPKN3F5RV4I2WPP6PXBQSZTIEVF2OZQIBY7PIQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:42.700160844 (526h26m42.571243879s) [noise=true , address=77.90.31.11:4556 , piece_id=2X3DL3V773MFFXYVQYRIYKXIRU4JZI4KELFIXZS3PUTBPZVTGKRA, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:42.928495633 (526h26m42.799578678s) [noise=true , address=77.90.31.19:4842 , piece_id=OGWSBYPGFHPWZ5VJISQ7A3PSYN7GGS6QVYINEAWBGAP4ADKVMPAA, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:43.041556590 (526h26m42.912639625s) [noise=true , address=77.90.31.12:4589 , piece_id=WRCB4QABBNPHWTPXEEO4SLB7MR3MT467XFOZTAWCCWG67ZAAK4GA, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:43.041576267 (526h26m42.912659302s) [seg_pos={0 0}, upload=461769]: Segment reached optimal threshold of 13 pieces. Dec 27 13:55:43.041779755 (526h26m42.91286279s) [noise=true , address=145.79.227.33:4869 , piece_id=MCLA62SXY6NLKAE5AJJQS4EKPY6NH6CO4HNSEW4HU5PMAALAEDUA, seg_pos={0 0}, upload=461769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 27 13:55:43.041785526 (526h26m42.912868561s) [noise=true , address=145.79.227.33:4869 , piece_id=MCLA62SXY6NLKAE5AJJQS4EKPY6NH6CO4HNSEW4HU5PMAALAEDUA, seg_pos={0 0}, upload=461769]: Long tail cancellation due to optimal threshold Dec 27 13:55:43.042105304 (526h26m42.91318837s) [noise=true , address=45.91.116.13:4153 , piece_id=VSAPME7HWTPXGRIHNZOOW7CHLHC55ELZUVRHNN7A7SHEVEEJEOYQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:36608->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:36608->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36608->45.91.116.13:4153: use of closed network connection Dec 27 13:55:43.042131284 (526h26m42.913214319s) [noise=true , address=45.91.116.13:4153 , piece_id=VSAPME7HWTPXGRIHNZOOW7CHLHC55ELZUVRHNN7A7SHEVEEJEOYQ, seg_pos={0 0}, upload=461769]: Long tail cancellation due to optimal threshold Dec 27 13:55:43.042255090 (526h26m42.913338125s) [noise=true , address=45.91.116.12:4225 , piece_id=NMARGNK6INEKMWLK2IQSTSMAVSE36E3T4WXMD6DEQMJKYTB6ZKXA, seg_pos={0 0}, upload=461769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:47484->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:47484->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47484->45.91.116.12:4225: use of closed network connection Dec 27 13:55:43.042264498 (526h26m42.913347533s) [noise=true , address=45.91.116.12:4225 , piece_id=NMARGNK6INEKMWLK2IQSTSMAVSE36E3T4WXMD6DEQMJKYTB6ZKXA, seg_pos={0 0}, upload=461769]: Long tail cancellation due to optimal threshold Dec 27 13:55:43.042283343 (526h26m42.913366388s) [seg_pos={0 0}, upload=461769]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:55:43.042387552 (526h26m42.913470597s) [upload=461769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:55:43.042394376 (526h26m42.913477411s) [upload=461769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:55:43.323639824 (526h26m43.194722859s) [upload=461769]: Done uploading segments... Dec 27 13:55:43.323658681 (526h26m43.194741716s) [upload=461769]: Upload finished. err: <nil> Duration: 5.892889533s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 34 out of 60 is failed (threshold=6) [....XXXXX..XXXXX....XX....X.X.X.X.X..XXX.XX.XX.X.XX.X.XXXXXX] duration: 5.001362655s trace: https://jaeger.dev.storj.io/trace/5fb73d2cb8877620 FAIL (2025-12-27 13:56:13 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001375519s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X..X...X.XXX....X...X...X...XX.......X....X..XX.X.XXXX..X..X] duration: 30.003147428s trace: https://jaeger.dev.storj.io/trace/61cbbcf7e1004bc3 FAIL (2025-12-27 13:56:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: WRCB4QABBNPHWTPXEEO4SLB7MR3MT467XFOZTAWCCWG67ZAAK4GA) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: 2X3DL3V773MFFXYVQYRIYKXIRU4JZI4KELFIXZS3PUTBPZVTGKRA) context deadline exceeded ======LOGS====== ing segments... Dec 27 13:55:39.506895724 (526h26m39.377978769s) [upload=461769]: Got next segment. Inline: false Dec 27 13:55:39.506937954 (526h26m39.378021s) [upload=461769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:55:39.506943695 (526h26m39.37802673s) [upload=461769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:55:39.746845108 (526h26m39.617928254s) [seg_pos={0 0}, upload=461769]: Begin upload segment... Dec 27 13:55:39.746972181 (526h26m39.618055226s) [seg_pos={0 0}, upload=461769]: Done begin upload segment. Dec 27 13:55:39.747039980 (526h26m39.618123025s) [noise=true , address=145.79.227.23:4546 , piece_id=SFSPRBAS5DZ7PGSPKN3F5RV4I2WPP6PXBQSZTIEVF2OZQIBY7PIQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.747055749 (526h26m39.618138794s) [noise=true , address=145.79.227.16:4912 , piece_id=2V6SNFMGJJF6PX7YA63PQUHCWEBEQQRXTIV5YP5IXPGOHL57DOWQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.747668517 (526h26m39.618751562s) [noise=true , address=145.79.227.28:4727 , piece_id=NSE2QHV2T375DF7JQO32WROCMIA3HUVBNBZZW4LCBCJ2K4CAL23Q, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.748154081 (526h26m39.619237136s) [noise=true , address=45.91.116.12:4225 , piece_id=NMARGNK6INEKMWLK2IQSTSMAVSE36E3T4WXMD6DEQMJKYTB6ZKXA, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.748539615 (526h26m39.61962266s) [noise=true , address=145.79.227.41:4085 , piece_id=4T7J4A56XHXQ7J5MP3PB5GUPL6ZQRYLTZ66IMD3GJVE4EGQY5JIA, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.748812355 (526h26m39.6198954s) [noise=true , address=77.90.31.12:4589 , piece_id=WRCB4QABBNPHWTPXEEO4SLB7MR3MT467XFOZTAWCCWG67ZAAK4GA, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.749151961 (526h26m39.620235036s) [noise=true , address=145.79.227.47:4267 , piece_id=TUZIXXBICDEZ64AI7XOVBBQRKM4WS7UVXOSFLGUXRZZXZJ5QFRZQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.749662393 (526h26m39.620745438s) [noise=true , address=145.79.227.33:4869 , piece_id=MCLA62SXY6NLKAE5AJJQS4EKPY6NH6CO4HNSEW4HU5PMAALAEDUA, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.750118381 (526h26m39.621201416s) [noise=true , address=145.79.227.27:4689 , piece_id=KCW5QW3O36SBVXA6BJV5DIRTS4HOYMQSAFRXCCCNHB3OPAXXFCFQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.750451205 (526h26m39.62153424s) [noise=true , address=145.79.227.37:4567 , piece_id=REVK53YXB7BT7T2IXTHM4NQBLC5LD2V6Q3AIX5I4O5Y2V5VQJDHQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.750760544 (526h26m39.62184358s) [noise=true , address=45.91.116.13:4153 , piece_id=VSAPME7HWTPXGRIHNZOOW7CHLHC55ELZUVRHNN7A7SHEVEEJEOYQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.751022002 (526h26m39.622105077s) [noise=true , address=145.79.227.29:4942 , piece_id=XQY6IDGGUO73AYWSQPJ5B2K724RWGVNDNRKPUBUKOGR7WPLBLWQQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.751446380 (526h26m39.622529415s) [noise=true , address=145.79.227.13:4804 , piece_id=LSHRLHBDQ5XYJCZ4XCOKY47EWU34SRRJWEORJS4PF4BTU7HA6DAA, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.751778563 (526h26m39.622861608s) [noise=true , address=77.90.31.11:4556 , piece_id=2X3DL3V773MFFXYVQYRIYKXIRU4JZI4KELFIXZS3PUTBPZVTGKRA, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.752151403 (526h26m39.623234438s) [noise=true , address=145.79.227.45:4231 , piece_id=ZI2FWBYUL4G7PCN5YZWAELM22AQP2O7Z6HML5NMGK5UJ3RSTEODQ, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:39.752460461 (526h26m39.623543496s) [noise=true , address=77.90.31.19:4842 , piece_id=OGWSBYPGFHPWZ5VJISQ7A3PSYN7GGS6QVYINEAWBGAP4ADKVMPAA, seg_pos={0 0}, upload=461769]: Uploading piece... Dec 27 13:55:41.454144697 (526h26m41.325227742s) [upload=461769]: Next returned nil segment Dec 27 13:55:41.454167932 (526h26m41.325250977s) [upload=461769]: Waiting for error group managing segments... Dec 27 13:55:41.738963230 (526h26m41.610046265s) [noise=true , address=145.79.227.37:4567 , piece_id=REVK53YXB7BT7T2IXTHM4NQBLC5LD2V6Q3AIX5I4O5Y2V5VQJDHQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.742975230 (526h26m41.614058265s) [noise=true , address=145.79.227.27:4689 , piece_id=KCW5QW3O36SBVXA6BJV5DIRTS4HOYMQSAFRXCCCNHB3OPAXXFCFQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.751149826 (526h26m41.622232871s) [noise=true , address=145.79.227.41:4085 , piece_id=4T7J4A56XHXQ7J5MP3PB5GUPL6ZQRYLTZ66IMD3GJVE4EGQY5JIA, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.751779916 (526h26m41.622862951s) [noise=true , address=145.79.227.45:4231 , piece_id=ZI2FWBYUL4G7PCN5YZWAELM22AQP2O7Z6HML5NMGK5UJ3RSTEODQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.755286142 (526h26m41.626369177s) [noise=true , address=145.79.227.16:4912 , piece_id=2V6SNFMGJJF6PX7YA63PQUHCWEBEQQRXTIV5YP5IXPGOHL57DOWQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.763271658 (526h26m41.634354693s) [noise=true , address=145.79.227.13:4804 , piece_id=LSHRLHBDQ5XYJCZ4XCOKY47EWU34SRRJWEORJS4PF4BTU7HA6DAA, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.764016025 (526h26m41.63509906s) [noise=true , address=145.79.227.28:4727 , piece_id=NSE2QHV2T375DF7JQO32WROCMIA3HUVBNBZZW4LCBCJ2K4CAL23Q, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.764095897 (526h26m41.635178932s) [noise=true , address=145.79.227.29:4942 , piece_id=XQY6IDGGUO73AYWSQPJ5B2K724RWGVNDNRKPUBUKOGR7WPLBLWQQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.778988115 (526h26m41.65007116s) [noise=true , address=145.79.227.47:4267 , piece_id=TUZIXXBICDEZ64AI7XOVBBQRKM4WS7UVXOSFLGUXRZZXZJ5QFRZQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:41.783665982 (526h26m41.654749027s) [noise=true , address=145.79.227.23:4546 , piece_id=SFSPRBAS5DZ7PGSPKN3F5RV4I2WPP6PXBQSZTIEVF2OZQIBY7PIQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:42.700160844 (526h26m42.571243879s) [noise=true , address=77.90.31.11:4556 , piece_id=2X3DL3V773MFFXYVQYRIYKXIRU4JZI4KELFIXZS3PUTBPZVTGKRA, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:42.928495633 (526h26m42.799578678s) [noise=true , address=77.90.31.19:4842 , piece_id=OGWSBYPGFHPWZ5VJISQ7A3PSYN7GGS6QVYINEAWBGAP4ADKVMPAA, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:43.041556590 (526h26m42.912639625s) [noise=true , address=77.90.31.12:4589 , piece_id=WRCB4QABBNPHWTPXEEO4SLB7MR3MT467XFOZTAWCCWG67ZAAK4GA, seg_pos={0 0}, upload=461769]: Done uploading piece. err: <nil> Dec 27 13:55:43.041576267 (526h26m42.912659302s) [seg_pos={0 0}, upload=461769]: Segment reached optimal threshold of 13 pieces. Dec 27 13:55:43.041779755 (526h26m42.91286279s) [noise=true , address=145.79.227.33:4869 , piece_id=MCLA62SXY6NLKAE5AJJQS4EKPY6NH6CO4HNSEW4HU5PMAALAEDUA, seg_pos={0 0}, upload=461769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 27 13:55:43.041785526 (526h26m42.912868561s) [noise=true , address=145.79.227.33:4869 , piece_id=MCLA62SXY6NLKAE5AJJQS4EKPY6NH6CO4HNSEW4HU5PMAALAEDUA, seg_pos={0 0}, upload=461769]: Long tail cancellation due to optimal threshold Dec 27 13:55:43.042105304 (526h26m42.91318837s) [noise=true , address=45.91.116.13:4153 , piece_id=VSAPME7HWTPXGRIHNZOOW7CHLHC55ELZUVRHNN7A7SHEVEEJEOYQ, seg_pos={0 0}, upload=461769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:36608->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:36608->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36608->45.91.116.13:4153: use of closed network connection Dec 27 13:55:43.042131284 (526h26m42.913214319s) [noise=true , address=45.91.116.13:4153 , piece_id=VSAPME7HWTPXGRIHNZOOW7CHLHC55ELZUVRHNN7A7SHEVEEJEOYQ, seg_pos={0 0}, upload=461769]: Long tail cancellation due to optimal threshold Dec 27 13:55:43.042255090 (526h26m42.913338125s) [noise=true , address=45.91.116.12:4225 , piece_id=NMARGNK6INEKMWLK2IQSTSMAVSE36E3T4WXMD6DEQMJKYTB6ZKXA, seg_pos={0 0}, upload=461769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:47484->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:47484->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47484->45.91.116.12:4225: use of closed network connection Dec 27 13:55:43.042264498 (526h26m42.913347533s) [noise=true , address=45.91.116.12:4225 , piece_id=NMARGNK6INEKMWLK2IQSTSMAVSE36E3T4WXMD6DEQMJKYTB6ZKXA, seg_pos={0 0}, upload=461769]: Long tail cancellation due to optimal threshold Dec 27 13:55:43.042283343 (526h26m42.913366388s) [seg_pos={0 0}, upload=461769]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:55:43.042387552 (526h26m42.913470597s) [upload=461769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:55:43.042394376 (526h26m42.913477411s) [upload=461769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:55:43.323639824 (526h26m43.194722859s) [upload=461769]: Done uploading segments... Dec 27 13:55:43.323658681 (526h26m43.194741716s) [upload=461769]: Upload finished. err: <nil> Duration: 30.003172215s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............X...X...X..........X.............XX............] duration: 42.976607783s trace: https://jaeger.dev.storj.io/trace/3ed1e657d506c4ee FAIL (2025-12-27 13:55:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, Piece ID: HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ) context deadline exceeded; piecestore: (Node ID: 12wXqePK1kK7iBo6wHcbqkRmkED8LGJ3rzQNJAWa14cHYhx1oyB, Piece ID: M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA) context deadline exceeded; piecestore: (Node ID: 12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx, Piece ID: ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ) context deadline exceeded; piecestore: (Node ID: 12UjFkFn2UEvr57LDorcijDCvkBkkKitgZEFx6Yqfk5gMStQvJG, Piece ID: SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ) context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.16:4739 , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.319646322 (526h14m34.190729357s) [noise=true , address=77.90.31.21:4899 , piece_id=SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.319933248 (526h14m34.191016283s) [noise=true , address=45.91.116.12:4234 , piece_id=Y2FIVYLZBQAYBO2TDGIZ4RDQHQJ3WOQH565A7ESIKANQQTQDFQZQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320400588 (526h14m34.191483633s) [noise=true , address=145.79.227.29:4952 , piece_id=VXZLF4OGRIR3MYQMMLKLTD235A55F4YTHB3D4YCVCQUL4FZTSFJQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320407341 (526h14m34.191490386s) [noise=true , address=145.79.227.46:4621 , piece_id=XZ4MKAZAGQKVEJJGSZ5QGT4AT3MFXXNMXICVFCI6SP3E4EVEHZGQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320675030 (526h14m34.191758075s) [noise=true , address=45.91.116.10:4113 , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320802022 (526h14m34.191885067s) [noise=true , address=77.90.31.11:4569 , piece_id=YC2Q3HPJDVY3NEDZNC2Z4KVN6S6DDDGRARASF7KALFAOQ335A7HA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.321169643 (526h14m34.192252678s) [noise=true , address=77.90.31.19:4869 , piece_id=HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.321443483 (526h14m34.192526518s) [noise=true , address=45.91.116.14:4202 , piece_id=M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.321731813 (526h14m34.192814848s) [noise=true , address=77.90.31.18:4819 , piece_id=YS4DXTHLGCPFOKKLBA2DK2BVOAPTQEGVOPBSRCLFV26SVP6YFHGA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.322221776 (526h14m34.193304811s) [noise=true , address=77.90.31.15:4708 , piece_id=3NX6E54XMZXQAQQE27LTWUSEF5GAIF2XOPQFFAP7LOA2PL7KDL3Q, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.322547776 (526h14m34.193630811s) [noise=true , address=145.79.227.40:4079 , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.322908093 (526h14m34.193991138s) [noise=true , address=145.79.227.29:4975 , piece_id=DAMNBVP7DRSW5OAGYRRP4K5SNJ2WEUT7IAXO77KWRHLQEMBNERZQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.323267968 (526h14m34.194351003s) [noise=true , address=145.79.227.30:4787 , piece_id=OB2OTY3ZEHIUFYM6FY7HIS7HQ353Y2HL6XU4476HXOD26QVA4DPQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.323618165 (526h14m34.19470121s) [noise=true , address=64.49.13.169:4710 , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.324084893 (526h14m34.195167928s) [noise=true , address=193.149.164.143:4694 , piece_id=ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:35.783540136 (526h14m35.654623181s) [upload=461595]: Next returned nil segment Dec 27 13:43:35.783555856 (526h14m35.654638891s) [upload=461595]: Waiting for error group managing segments... Dec 27 13:43:36.192287250 (526h14m36.063370296s) [noise=true , address=145.79.227.29:4952 , piece_id=VXZLF4OGRIR3MYQMMLKLTD235A55F4YTHB3D4YCVCQUL4FZTSFJQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.200200289 (526h14m36.071283334s) [noise=true , address=145.79.227.46:4621 , piece_id=XZ4MKAZAGQKVEJJGSZ5QGT4AT3MFXXNMXICVFCI6SP3E4EVEHZGQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.238087532 (526h14m36.109170577s) [noise=true , address=145.79.227.29:4975 , piece_id=DAMNBVP7DRSW5OAGYRRP4K5SNJ2WEUT7IAXO77KWRHLQEMBNERZQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.379631346 (526h14m36.250714391s) [noise=true , address=77.90.31.19:4869 , piece_id=HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.382945286 (526h14m36.254028321s) [noise=true , address=145.79.227.34:4909 , piece_id=Z6RF5PSOGOZMOXPQYXHMXGFXIELVXXVU4NXXRTAQCIELQXLMVXBQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.400290103 (526h14m36.271373138s) [noise=true , address=77.90.31.18:4819 , piece_id=YS4DXTHLGCPFOKKLBA2DK2BVOAPTQEGVOPBSRCLFV26SVP6YFHGA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.436707208 (526h14m36.307790243s) [noise=true , address=45.91.116.14:4202 , piece_id=M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.487651475 (526h14m36.35873451s) [noise=true , address=45.91.116.12:4234 , piece_id=Y2FIVYLZBQAYBO2TDGIZ4RDQHQJ3WOQH565A7ESIKANQQTQDFQZQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.500850769 (526h14m36.371933814s) [noise=true , address=77.90.31.11:4569 , piece_id=YC2Q3HPJDVY3NEDZNC2Z4KVN6S6DDDGRARASF7KALFAOQ335A7HA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.518828551 (526h14m36.389911586s) [noise=true , address=77.90.31.15:4708 , piece_id=3NX6E54XMZXQAQQE27LTWUSEF5GAIF2XOPQFFAP7LOA2PL7KDL3Q, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.051607243 (526h14m36.922690289s) [noise=true , address=145.79.227.44:4228 , piece_id=6SAYEAOTTCOUFPU75C7YRIULWJGXL4PK2YKCKFB47EUBWMRQQQOQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.096826909 (526h14m36.967909954s) [noise=true , address=145.79.227.30:4787 , piece_id=OB2OTY3ZEHIUFYM6FY7HIS7HQ353Y2HL6XU4476HXOD26QVA4DPQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.140738995 (526h14m37.01182203s) [noise=true , address=77.90.31.21:4899 , piece_id=SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.323613990 (526h14m37.194697035s) [noise=true , address=193.149.164.143:4694 , piece_id=ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.323649618 (526h14m37.194732663s) [seg_pos={0 0}, upload=461595]: Segment reached optimal threshold of 14 pieces. Dec 27 13:43:37.323852865 (526h14m37.1949359s) [noise=true , address=45.91.116.10:4113 , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12puRMny4nbgv2by4tAbXcBBSgG2d6BfwcDDRDcFzEdjSVUJxtd): protocol: expected piece hash; context canceled; EOF Dec 27 13:43:37.323860349 (526h14m37.194943384s) [noise=true , address=45.91.116.10:4113 , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324005135 (526h14m37.1950882s) [noise=true , address=145.79.227.40:4079 , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RYKzn5zVSmxVbWwnBdX8oqtG1WE2MhPeEiWLf5dZeFAut6s2Y): protocol: expected piece hash; context canceled; EOF Dec 27 13:43:37.324022659 (526h14m37.195105704s) [noise=true , address=145.79.227.40:4079 , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324143378 (526h14m37.195226413s) [noise=true , address=77.90.31.16:4739 , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129B3N7yjJvwCzoVLTqWsPbcLQ5TQUyAzg1RCkbSAVKPWYTG8eb): protocol: expected piece hash; context canceled; EOF Dec 27 13:43:37.324154520 (526h14m37.195237555s) [noise=true , address=77.90.31.16:4739 , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324213442 (526h14m37.195296487s) [noise=true , address=64.49.13.169:4710 , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection Dec 27 13:43:37.324249360 (526h14m37.195332405s) [noise=true , address=64.49.13.169:4710 , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324275360 (526h14m37.195358405s) [seg_pos={0 0}, upload=461595]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 13:43:37.324374018 (526h14m37.195457053s) [upload=461595]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:43:37.324381291 (526h14m37.195464337s) [upload=461595]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:43:37.679036406 (526h14m37.550119451s) [upload=461595]: Done uploading segments... Dec 27 13:43:37.679056985 (526h14m37.55014003s) [upload=461595]: Upload finished. err: <nil> Duration: 42.976624524s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [.....XXXXX..XXXXX....XX....X.X.X.X.X..XXX.XX.XX.X.XX.X.XXXXX] duration: 5.001866743s trace: https://jaeger.dev.storj.io/trace/334c62186ce06a85 FAIL (2025-12-27 13:55:31 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00189168s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X..X...X.XXX....X...X...X...XX.......X....X..XX.X.XXXX..X..] duration: 5.965613751s trace: https://jaeger.dev.storj.io/trace/6a91a73558fd48b8 FAIL (2025-12-27 13:55:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== egments... Dec 27 13:53:37.731749182 (526h24m37.602832217s) [upload=461738]: Got next segment. Inline: false Dec 27 13:53:37.731797104 (526h24m37.602880139s) [upload=461738]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:53:37.731801142 (526h24m37.602884177s) [upload=461738]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:53:37.979178972 (526h24m37.850262007s) [seg_pos={0 0}, upload=461738]: Begin upload segment... Dec 27 13:53:37.979241551 (526h24m37.850324586s) [seg_pos={0 0}, upload=461738]: Done begin upload segment. Dec 27 13:53:37.979264555 (526h24m37.8503476s) [noise=true , address=145.79.227.4:4187 , piece_id=6ZP3H5LSTA277NHYNRKHVUI3GXR53DSCYJDH2FXVPLQ46BDJTXCQ, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:37.979281617 (526h24m37.850364663s) [noise=true , address=45.91.116.5:4298 , piece_id=4YA3J7U3UKHSPDO2I44AIQ22RVPISYUIDFQVH56FANXNWV27SGAQ, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:37.979451561 (526h24m37.850534606s) [noise=true , address=145.79.227.29:4943 , piece_id=42SBBOV57XGWGT4CE4X4YXZGLGH3JAJ5K7CRF5QQ3ZWMLVNFZCGA, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:37.979907429 (526h24m37.850990474s) [noise=true , address=45.91.116.14:4189 , piece_id=FSN3WGFV6MVAPS64SH72DW3HQLLA5DWDSJK4LMEF6VYR62YOCMQQ, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:37.980327860 (526h24m37.851410905s) [noise=true , address=45.91.116.4:4937 , piece_id=IEHFN6JGIQW34G7IURPY3VZYILWN6LY2X7COKLFDZKMEO6WXS4XA, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:37.980561855 (526h24m37.85164489s) [noise=true , address=45.91.116.6:4973 , piece_id=JHXQTGS5ZK4HFNHCQMCOT4ZWEIDVROIZK7B5RJ5VVHCGNTE6MMRQ, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:37.981010670 (526h24m37.852093715s) [noise=true , address=145.79.227.45:4231 , piece_id=2NT4YO5R5WR2SW6XGCVQJ3CTSMPSLA64QW4PAHZ72ZHK5HEU6SPQ, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:37.981293538 (526h24m37.852376573s) [noise=true , address=45.91.116.8:4046 , piece_id=TLE7NOEIS747XZ3Z724GBISGZHBIER3JQRY66DPID55XVQ57ZU5Q, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:37.981567590 (526h24m37.852650625s) [noise=true , address=45.91.116.12:4226 , piece_id=GRNEZRU2ST5NLP5B37GEVBKZF76YNJYI4BV2IOHIXATWTQ2F2Q2A, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:37.981879364 (526h24m37.852962399s) [noise=true , address=45.91.116.10:4082 , piece_id=53FIQMUADW73MC7T5BO4QFBEXAYO4FB5KPYE65QE2BASJZF6B2LQ, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:37.982273143 (526h24m37.853356178s) [noise=true , address=145.79.227.17:4972 , piece_id=DEI2KTZI2AADI2BTB3SUTM24PEY44HCAURX7RBBXU7VZFK4EMS4A, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:37.982648629 (526h24m37.853731664s) [noise=true , address=145.79.227.47:4265 , piece_id=3IU4FJTXFR33UF6EQKDKQ3TMV7BL4CXMC6QZYB5RQMWEWLNKRQNA, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:37.983089719 (526h24m37.854172764s) [noise=true , address=145.79.227.39:4014 , piece_id=O27D7BJLIMTATT2CPPBOWVZAY7Z4XM5J7QNJBEFOZX5VIPOKEI5Q, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:37.983419777 (526h24m37.854502822s) [noise=true , address=145.79.227.26:4655 , piece_id=YSPCEP4SNAHGXLUS6EJWEAFH7RFGUZQ75BD62FBHDPNYNG5N34JA, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:37.983808368 (526h24m37.854891403s) [noise=true , address=145.79.227.42:4122 , piece_id=UWCCJ6OSPQUX435UEYQ7FSN5PLWY74Y6DQM2ZE6BUGGBXUJKH6WA, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:37.985006178 (526h24m37.856089213s) [noise=true , address=145.79.227.52:4446 , piece_id=YX4RYQ64OMLM5OYGIR62TILRCL2JP2PFDW275SK3D2X24EL7HGAA, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:39.607227113 (526h24m39.478310148s) [upload=461738]: Next returned nil segment Dec 27 13:53:39.607242942 (526h24m39.478325987s) [upload=461738]: Waiting for error group managing segments... Dec 27 13:53:39.979288701 (526h24m39.850371736s) [noise=true , address=145.79.227.26:4655 , piece_id=YSPCEP4SNAHGXLUS6EJWEAFH7RFGUZQ75BD62FBHDPNYNG5N34JA, seg_pos={0 0}, upload=461738]: Done uploading piece. err: <nil> Dec 27 13:53:39.991948717 (526h24m39.863031752s) [noise=true , address=145.79.227.42:4122 , piece_id=UWCCJ6OSPQUX435UEYQ7FSN5PLWY74Y6DQM2ZE6BUGGBXUJKH6WA, seg_pos={0 0}, upload=461738]: Done uploading piece. err: <nil> Dec 27 13:53:39.992192942 (526h24m39.863275977s) [noise=true , address=145.79.227.39:4014 , piece_id=O27D7BJLIMTATT2CPPBOWVZAY7Z4XM5J7QNJBEFOZX5VIPOKEI5Q, seg_pos={0 0}, upload=461738]: Done uploading piece. err: <nil> Dec 27 13:53:39.995436820 (526h24m39.866519865s) [noise=true , address=145.79.227.45:4231 , piece_id=2NT4YO5R5WR2SW6XGCVQJ3CTSMPSLA64QW4PAHZ72ZHK5HEU6SPQ, seg_pos={0 0}, upload=461738]: Done uploading piece. err: <nil> Dec 27 13:53:39.995998078 (526h24m39.867081114s) [noise=true , address=145.79.227.17:4972 , piece_id=DEI2KTZI2AADI2BTB3SUTM24PEY44HCAURX7RBBXU7VZFK4EMS4A, seg_pos={0 0}, upload=461738]: Done uploading piece. err: <nil> Dec 27 13:53:40.001367613 (526h24m39.872450678s) [noise=true , address=145.79.227.47:4265 , piece_id=3IU4FJTXFR33UF6EQKDKQ3TMV7BL4CXMC6QZYB5RQMWEWLNKRQNA, seg_pos={0 0}, upload=461738]: Done uploading piece. err: <nil> Dec 27 13:53:40.001643679 (526h24m39.872726754s) [noise=true , address=145.79.227.4:4187 , piece_id=6ZP3H5LSTA277NHYNRKHVUI3GXR53DSCYJDH2FXVPLQ46BDJTXCQ, seg_pos={0 0}, upload=461738]: Done uploading piece. err: <nil> Dec 27 13:53:40.017537273 (526h24m39.888620308s) [noise=true , address=145.79.227.29:4943 , piece_id=42SBBOV57XGWGT4CE4X4YXZGLGH3JAJ5K7CRF5QQ3ZWMLVNFZCGA, seg_pos={0 0}, upload=461738]: Done uploading piece. err: <nil> Dec 27 13:53:40.019145205 (526h24m39.89022825s) [noise=true , address=145.79.227.52:4446 , piece_id=YX4RYQ64OMLM5OYGIR62TILRCL2JP2PFDW275SK3D2X24EL7HGAA, seg_pos={0 0}, upload=461738]: Done uploading piece. err: <nil> Dec 27 13:53:40.247411182 (526h24m40.118494227s) [noise=true , address=45.91.116.5:4298 , piece_id=4YA3J7U3UKHSPDO2I44AIQ22RVPISYUIDFQVH56FANXNWV27SGAQ, seg_pos={0 0}, upload=461738]: Done uploading piece. err: <nil> Dec 27 13:53:40.265252304 (526h24m40.136335339s) [noise=true , address=45.91.116.14:4189 , piece_id=FSN3WGFV6MVAPS64SH72DW3HQLLA5DWDSJK4LMEF6VYR62YOCMQQ, seg_pos={0 0}, upload=461738]: Done uploading piece. err: <nil> Dec 27 13:53:40.275178465 (526h24m40.14626151s) [noise=true , address=45.91.116.12:4226 , piece_id=GRNEZRU2ST5NLP5B37GEVBKZF76YNJYI4BV2IOHIXATWTQ2F2Q2A, seg_pos={0 0}, upload=461738]: Done uploading piece. err: <nil> Dec 27 13:53:40.445530043 (526h24m40.316613088s) [noise=true , address=45.91.116.6:4973 , piece_id=JHXQTGS5ZK4HFNHCQMCOT4ZWEIDVROIZK7B5RJ5VVHCGNTE6MMRQ, seg_pos={0 0}, upload=461738]: Done uploading piece. err: <nil> Dec 27 13:53:40.445550182 (526h24m40.316633227s) [seg_pos={0 0}, upload=461738]: Segment reached optimal threshold of 13 pieces. Dec 27 13:53:40.445786192 (526h24m40.316869227s) [noise=true , address=45.91.116.10:4082 , piece_id=53FIQMUADW73MC7T5BO4QFBEXAYO4FB5KPYE65QE2BASJZF6B2LQ, seg_pos={0 0}, upload=461738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:46986->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:46986->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46986->45.91.116.10:4082: use of closed network connection Dec 27 13:53:40.445793215 (526h24m40.31687625s) [noise=true , address=45.91.116.10:4082 , piece_id=53FIQMUADW73MC7T5BO4QFBEXAYO4FB5KPYE65QE2BASJZF6B2LQ, seg_pos={0 0}, upload=461738]: Long tail cancellation due to optimal threshold Dec 27 13:53:40.446068259 (526h24m40.317151294s) [noise=true , address=45.91.116.4:4937 , piece_id=IEHFN6JGIQW34G7IURPY3VZYILWN6LY2X7COKLFDZKMEO6WXS4XA, seg_pos={0 0}, upload=461738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 27 13:53:40.446077987 (526h24m40.317161022s) [noise=true , address=45.91.116.4:4937 , piece_id=IEHFN6JGIQW34G7IURPY3VZYILWN6LY2X7COKLFDZKMEO6WXS4XA, seg_pos={0 0}, upload=461738]: Long tail cancellation due to optimal threshold Dec 27 13:53:40.446133633 (526h24m40.317216668s) [noise=true , address=45.91.116.8:4046 , piece_id=TLE7NOEIS747XZ3Z724GBISGZHBIER3JQRY66DPID55XVQ57ZU5Q, seg_pos={0 0}, upload=461738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:47622->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:47622->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47622->45.91.116.8:4046: use of closed network connection Dec 27 13:53:40.446138813 (526h24m40.317221848s) [noise=true , address=45.91.116.8:4046 , piece_id=TLE7NOEIS747XZ3Z724GBISGZHBIER3JQRY66DPID55XVQ57ZU5Q, seg_pos={0 0}, upload=461738]: Long tail cancellation due to optimal threshold Dec 27 13:53:40.446153611 (526h24m40.317236656s) [seg_pos={0 0}, upload=461738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:53:40.446228083 (526h24m40.317311118s) [upload=461738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:53:40.446232631 (526h24m40.317315666s) [upload=461738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:53:40.739780236 (526h24m40.610863281s) [upload=461738]: Done uploading segments... Dec 27 13:53:40.739796758 (526h24m40.610879793s) [upload=461738]: Upload finished. err: <nil> Duration: 5.965646525s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 32 out of 60 is failed (threshold=6) [......XXXXX..XXXXX....XX....X.X.X.X.X..XXX.XX.XX.X.XX.X.XXXX] duration: 5.001930818s trace: https://jaeger.dev.storj.io/trace/6408b9310b65bd22 FAIL (2025-12-27 13:55:02 UTC) orbiter warning: context deadline exceeded Duration: 5.001956556s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............X...X...X..........X.............XX...........] duration: 53.571741511s trace: https://jaeger.dev.storj.io/trace/3ed1e657d506c4ee FAIL (2025-12-27 13:54:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, Piece ID: HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ) context deadline exceeded; piecestore: (Node ID: 12wXqePK1kK7iBo6wHcbqkRmkED8LGJ3rzQNJAWa14cHYhx1oyB, Piece ID: M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA) context deadline exceeded; piecestore: (Node ID: 12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx, Piece ID: ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ) context deadline exceeded; piecestore: (Node ID: 12UjFkFn2UEvr57LDorcijDCvkBkkKitgZEFx6Yqfk5gMStQvJG, Piece ID: SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ) context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.16:4739 , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.319646322 (526h14m34.190729357s) [noise=true , address=77.90.31.21:4899 , piece_id=SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.319933248 (526h14m34.191016283s) [noise=true , address=45.91.116.12:4234 , piece_id=Y2FIVYLZBQAYBO2TDGIZ4RDQHQJ3WOQH565A7ESIKANQQTQDFQZQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320400588 (526h14m34.191483633s) [noise=true , address=145.79.227.29:4952 , piece_id=VXZLF4OGRIR3MYQMMLKLTD235A55F4YTHB3D4YCVCQUL4FZTSFJQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320407341 (526h14m34.191490386s) [noise=true , address=145.79.227.46:4621 , piece_id=XZ4MKAZAGQKVEJJGSZ5QGT4AT3MFXXNMXICVFCI6SP3E4EVEHZGQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320675030 (526h14m34.191758075s) [noise=true , address=45.91.116.10:4113 , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320802022 (526h14m34.191885067s) [noise=true , address=77.90.31.11:4569 , piece_id=YC2Q3HPJDVY3NEDZNC2Z4KVN6S6DDDGRARASF7KALFAOQ335A7HA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.321169643 (526h14m34.192252678s) [noise=true , address=77.90.31.19:4869 , piece_id=HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.321443483 (526h14m34.192526518s) [noise=true , address=45.91.116.14:4202 , piece_id=M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.321731813 (526h14m34.192814848s) [noise=true , address=77.90.31.18:4819 , piece_id=YS4DXTHLGCPFOKKLBA2DK2BVOAPTQEGVOPBSRCLFV26SVP6YFHGA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.322221776 (526h14m34.193304811s) [noise=true , address=77.90.31.15:4708 , piece_id=3NX6E54XMZXQAQQE27LTWUSEF5GAIF2XOPQFFAP7LOA2PL7KDL3Q, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.322547776 (526h14m34.193630811s) [noise=true , address=145.79.227.40:4079 , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.322908093 (526h14m34.193991138s) [noise=true , address=145.79.227.29:4975 , piece_id=DAMNBVP7DRSW5OAGYRRP4K5SNJ2WEUT7IAXO77KWRHLQEMBNERZQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.323267968 (526h14m34.194351003s) [noise=true , address=145.79.227.30:4787 , piece_id=OB2OTY3ZEHIUFYM6FY7HIS7HQ353Y2HL6XU4476HXOD26QVA4DPQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.323618165 (526h14m34.19470121s) [noise=true , address=64.49.13.169:4710 , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.324084893 (526h14m34.195167928s) [noise=true , address=193.149.164.143:4694 , piece_id=ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:35.783540136 (526h14m35.654623181s) [upload=461595]: Next returned nil segment Dec 27 13:43:35.783555856 (526h14m35.654638891s) [upload=461595]: Waiting for error group managing segments... Dec 27 13:43:36.192287250 (526h14m36.063370296s) [noise=true , address=145.79.227.29:4952 , piece_id=VXZLF4OGRIR3MYQMMLKLTD235A55F4YTHB3D4YCVCQUL4FZTSFJQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.200200289 (526h14m36.071283334s) [noise=true , address=145.79.227.46:4621 , piece_id=XZ4MKAZAGQKVEJJGSZ5QGT4AT3MFXXNMXICVFCI6SP3E4EVEHZGQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.238087532 (526h14m36.109170577s) [noise=true , address=145.79.227.29:4975 , piece_id=DAMNBVP7DRSW5OAGYRRP4K5SNJ2WEUT7IAXO77KWRHLQEMBNERZQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.379631346 (526h14m36.250714391s) [noise=true , address=77.90.31.19:4869 , piece_id=HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.382945286 (526h14m36.254028321s) [noise=true , address=145.79.227.34:4909 , piece_id=Z6RF5PSOGOZMOXPQYXHMXGFXIELVXXVU4NXXRTAQCIELQXLMVXBQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.400290103 (526h14m36.271373138s) [noise=true , address=77.90.31.18:4819 , piece_id=YS4DXTHLGCPFOKKLBA2DK2BVOAPTQEGVOPBSRCLFV26SVP6YFHGA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.436707208 (526h14m36.307790243s) [noise=true , address=45.91.116.14:4202 , piece_id=M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.487651475 (526h14m36.35873451s) [noise=true , address=45.91.116.12:4234 , piece_id=Y2FIVYLZBQAYBO2TDGIZ4RDQHQJ3WOQH565A7ESIKANQQTQDFQZQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.500850769 (526h14m36.371933814s) [noise=true , address=77.90.31.11:4569 , piece_id=YC2Q3HPJDVY3NEDZNC2Z4KVN6S6DDDGRARASF7KALFAOQ335A7HA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.518828551 (526h14m36.389911586s) [noise=true , address=77.90.31.15:4708 , piece_id=3NX6E54XMZXQAQQE27LTWUSEF5GAIF2XOPQFFAP7LOA2PL7KDL3Q, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.051607243 (526h14m36.922690289s) [noise=true , address=145.79.227.44:4228 , piece_id=6SAYEAOTTCOUFPU75C7YRIULWJGXL4PK2YKCKFB47EUBWMRQQQOQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.096826909 (526h14m36.967909954s) [noise=true , address=145.79.227.30:4787 , piece_id=OB2OTY3ZEHIUFYM6FY7HIS7HQ353Y2HL6XU4476HXOD26QVA4DPQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.140738995 (526h14m37.01182203s) [noise=true , address=77.90.31.21:4899 , piece_id=SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.323613990 (526h14m37.194697035s) [noise=true , address=193.149.164.143:4694 , piece_id=ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.323649618 (526h14m37.194732663s) [seg_pos={0 0}, upload=461595]: Segment reached optimal threshold of 14 pieces. Dec 27 13:43:37.323852865 (526h14m37.1949359s) [noise=true , address=45.91.116.10:4113 , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12puRMny4nbgv2by4tAbXcBBSgG2d6BfwcDDRDcFzEdjSVUJxtd): protocol: expected piece hash; context canceled; EOF Dec 27 13:43:37.323860349 (526h14m37.194943384s) [noise=true , address=45.91.116.10:4113 , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324005135 (526h14m37.1950882s) [noise=true , address=145.79.227.40:4079 , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RYKzn5zVSmxVbWwnBdX8oqtG1WE2MhPeEiWLf5dZeFAut6s2Y): protocol: expected piece hash; context canceled; EOF Dec 27 13:43:37.324022659 (526h14m37.195105704s) [noise=true , address=145.79.227.40:4079 , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324143378 (526h14m37.195226413s) [noise=true , address=77.90.31.16:4739 , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129B3N7yjJvwCzoVLTqWsPbcLQ5TQUyAzg1RCkbSAVKPWYTG8eb): protocol: expected piece hash; context canceled; EOF Dec 27 13:43:37.324154520 (526h14m37.195237555s) [noise=true , address=77.90.31.16:4739 , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324213442 (526h14m37.195296487s) [noise=true , address=64.49.13.169:4710 , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection Dec 27 13:43:37.324249360 (526h14m37.195332405s) [noise=true , address=64.49.13.169:4710 , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324275360 (526h14m37.195358405s) [seg_pos={0 0}, upload=461595]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 13:43:37.324374018 (526h14m37.195457053s) [upload=461595]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:43:37.324381291 (526h14m37.195464337s) [upload=461595]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:43:37.679036406 (526h14m37.550119451s) [upload=461595]: Done uploading segments... Dec 27 13:43:37.679056985 (526h14m37.55014003s) [upload=461595]: Upload finished. err: <nil> Duration: 53.571765977s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..X..X...X.XXX....X...X...X...XX.......X....X..XX.X.XXXX..X.] duration: 5.880765377s trace: https://jaeger.dev.storj.io/trace/6a91a73558fd48b8 FAIL (2025-12-27 13:54:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== egments... Dec 27 13:53:37.731749182 (526h24m37.602832217s) [upload=461738]: Got next segment. Inline: false Dec 27 13:53:37.731797104 (526h24m37.602880139s) [upload=461738]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:53:37.731801142 (526h24m37.602884177s) [upload=461738]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:53:37.979178972 (526h24m37.850262007s) [seg_pos={0 0}, upload=461738]: Begin upload segment... Dec 27 13:53:37.979241551 (526h24m37.850324586s) [seg_pos={0 0}, upload=461738]: Done begin upload segment. Dec 27 13:53:37.979264555 (526h24m37.8503476s) [noise=true , address=145.79.227.4:4187 , piece_id=6ZP3H5LSTA277NHYNRKHVUI3GXR53DSCYJDH2FXVPLQ46BDJTXCQ, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:37.979281617 (526h24m37.850364663s) [noise=true , address=45.91.116.5:4298 , piece_id=4YA3J7U3UKHSPDO2I44AIQ22RVPISYUIDFQVH56FANXNWV27SGAQ, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:37.979451561 (526h24m37.850534606s) [noise=true , address=145.79.227.29:4943 , piece_id=42SBBOV57XGWGT4CE4X4YXZGLGH3JAJ5K7CRF5QQ3ZWMLVNFZCGA, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:37.979907429 (526h24m37.850990474s) [noise=true , address=45.91.116.14:4189 , piece_id=FSN3WGFV6MVAPS64SH72DW3HQLLA5DWDSJK4LMEF6VYR62YOCMQQ, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:37.980327860 (526h24m37.851410905s) [noise=true , address=45.91.116.4:4937 , piece_id=IEHFN6JGIQW34G7IURPY3VZYILWN6LY2X7COKLFDZKMEO6WXS4XA, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:37.980561855 (526h24m37.85164489s) [noise=true , address=45.91.116.6:4973 , piece_id=JHXQTGS5ZK4HFNHCQMCOT4ZWEIDVROIZK7B5RJ5VVHCGNTE6MMRQ, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:37.981010670 (526h24m37.852093715s) [noise=true , address=145.79.227.45:4231 , piece_id=2NT4YO5R5WR2SW6XGCVQJ3CTSMPSLA64QW4PAHZ72ZHK5HEU6SPQ, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:37.981293538 (526h24m37.852376573s) [noise=true , address=45.91.116.8:4046 , piece_id=TLE7NOEIS747XZ3Z724GBISGZHBIER3JQRY66DPID55XVQ57ZU5Q, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:37.981567590 (526h24m37.852650625s) [noise=true , address=45.91.116.12:4226 , piece_id=GRNEZRU2ST5NLP5B37GEVBKZF76YNJYI4BV2IOHIXATWTQ2F2Q2A, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:37.981879364 (526h24m37.852962399s) [noise=true , address=45.91.116.10:4082 , piece_id=53FIQMUADW73MC7T5BO4QFBEXAYO4FB5KPYE65QE2BASJZF6B2LQ, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:37.982273143 (526h24m37.853356178s) [noise=true , address=145.79.227.17:4972 , piece_id=DEI2KTZI2AADI2BTB3SUTM24PEY44HCAURX7RBBXU7VZFK4EMS4A, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:37.982648629 (526h24m37.853731664s) [noise=true , address=145.79.227.47:4265 , piece_id=3IU4FJTXFR33UF6EQKDKQ3TMV7BL4CXMC6QZYB5RQMWEWLNKRQNA, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:37.983089719 (526h24m37.854172764s) [noise=true , address=145.79.227.39:4014 , piece_id=O27D7BJLIMTATT2CPPBOWVZAY7Z4XM5J7QNJBEFOZX5VIPOKEI5Q, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:37.983419777 (526h24m37.854502822s) [noise=true , address=145.79.227.26:4655 , piece_id=YSPCEP4SNAHGXLUS6EJWEAFH7RFGUZQ75BD62FBHDPNYNG5N34JA, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:37.983808368 (526h24m37.854891403s) [noise=true , address=145.79.227.42:4122 , piece_id=UWCCJ6OSPQUX435UEYQ7FSN5PLWY74Y6DQM2ZE6BUGGBXUJKH6WA, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:37.985006178 (526h24m37.856089213s) [noise=true , address=145.79.227.52:4446 , piece_id=YX4RYQ64OMLM5OYGIR62TILRCL2JP2PFDW275SK3D2X24EL7HGAA, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:39.607227113 (526h24m39.478310148s) [upload=461738]: Next returned nil segment Dec 27 13:53:39.607242942 (526h24m39.478325987s) [upload=461738]: Waiting for error group managing segments... Dec 27 13:53:39.979288701 (526h24m39.850371736s) [noise=true , address=145.79.227.26:4655 , piece_id=YSPCEP4SNAHGXLUS6EJWEAFH7RFGUZQ75BD62FBHDPNYNG5N34JA, seg_pos={0 0}, upload=461738]: Done uploading piece. err: <nil> Dec 27 13:53:39.991948717 (526h24m39.863031752s) [noise=true , address=145.79.227.42:4122 , piece_id=UWCCJ6OSPQUX435UEYQ7FSN5PLWY74Y6DQM2ZE6BUGGBXUJKH6WA, seg_pos={0 0}, upload=461738]: Done uploading piece. err: <nil> Dec 27 13:53:39.992192942 (526h24m39.863275977s) [noise=true , address=145.79.227.39:4014 , piece_id=O27D7BJLIMTATT2CPPBOWVZAY7Z4XM5J7QNJBEFOZX5VIPOKEI5Q, seg_pos={0 0}, upload=461738]: Done uploading piece. err: <nil> Dec 27 13:53:39.995436820 (526h24m39.866519865s) [noise=true , address=145.79.227.45:4231 , piece_id=2NT4YO5R5WR2SW6XGCVQJ3CTSMPSLA64QW4PAHZ72ZHK5HEU6SPQ, seg_pos={0 0}, upload=461738]: Done uploading piece. err: <nil> Dec 27 13:53:39.995998078 (526h24m39.867081114s) [noise=true , address=145.79.227.17:4972 , piece_id=DEI2KTZI2AADI2BTB3SUTM24PEY44HCAURX7RBBXU7VZFK4EMS4A, seg_pos={0 0}, upload=461738]: Done uploading piece. err: <nil> Dec 27 13:53:40.001367613 (526h24m39.872450678s) [noise=true , address=145.79.227.47:4265 , piece_id=3IU4FJTXFR33UF6EQKDKQ3TMV7BL4CXMC6QZYB5RQMWEWLNKRQNA, seg_pos={0 0}, upload=461738]: Done uploading piece. err: <nil> Dec 27 13:53:40.001643679 (526h24m39.872726754s) [noise=true , address=145.79.227.4:4187 , piece_id=6ZP3H5LSTA277NHYNRKHVUI3GXR53DSCYJDH2FXVPLQ46BDJTXCQ, seg_pos={0 0}, upload=461738]: Done uploading piece. err: <nil> Dec 27 13:53:40.017537273 (526h24m39.888620308s) [noise=true , address=145.79.227.29:4943 , piece_id=42SBBOV57XGWGT4CE4X4YXZGLGH3JAJ5K7CRF5QQ3ZWMLVNFZCGA, seg_pos={0 0}, upload=461738]: Done uploading piece. err: <nil> Dec 27 13:53:40.019145205 (526h24m39.89022825s) [noise=true , address=145.79.227.52:4446 , piece_id=YX4RYQ64OMLM5OYGIR62TILRCL2JP2PFDW275SK3D2X24EL7HGAA, seg_pos={0 0}, upload=461738]: Done uploading piece. err: <nil> Dec 27 13:53:40.247411182 (526h24m40.118494227s) [noise=true , address=45.91.116.5:4298 , piece_id=4YA3J7U3UKHSPDO2I44AIQ22RVPISYUIDFQVH56FANXNWV27SGAQ, seg_pos={0 0}, upload=461738]: Done uploading piece. err: <nil> Dec 27 13:53:40.265252304 (526h24m40.136335339s) [noise=true , address=45.91.116.14:4189 , piece_id=FSN3WGFV6MVAPS64SH72DW3HQLLA5DWDSJK4LMEF6VYR62YOCMQQ, seg_pos={0 0}, upload=461738]: Done uploading piece. err: <nil> Dec 27 13:53:40.275178465 (526h24m40.14626151s) [noise=true , address=45.91.116.12:4226 , piece_id=GRNEZRU2ST5NLP5B37GEVBKZF76YNJYI4BV2IOHIXATWTQ2F2Q2A, seg_pos={0 0}, upload=461738]: Done uploading piece. err: <nil> Dec 27 13:53:40.445530043 (526h24m40.316613088s) [noise=true , address=45.91.116.6:4973 , piece_id=JHXQTGS5ZK4HFNHCQMCOT4ZWEIDVROIZK7B5RJ5VVHCGNTE6MMRQ, seg_pos={0 0}, upload=461738]: Done uploading piece. err: <nil> Dec 27 13:53:40.445550182 (526h24m40.316633227s) [seg_pos={0 0}, upload=461738]: Segment reached optimal threshold of 13 pieces. Dec 27 13:53:40.445786192 (526h24m40.316869227s) [noise=true , address=45.91.116.10:4082 , piece_id=53FIQMUADW73MC7T5BO4QFBEXAYO4FB5KPYE65QE2BASJZF6B2LQ, seg_pos={0 0}, upload=461738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:46986->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:46986->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46986->45.91.116.10:4082: use of closed network connection Dec 27 13:53:40.445793215 (526h24m40.31687625s) [noise=true , address=45.91.116.10:4082 , piece_id=53FIQMUADW73MC7T5BO4QFBEXAYO4FB5KPYE65QE2BASJZF6B2LQ, seg_pos={0 0}, upload=461738]: Long tail cancellation due to optimal threshold Dec 27 13:53:40.446068259 (526h24m40.317151294s) [noise=true , address=45.91.116.4:4937 , piece_id=IEHFN6JGIQW34G7IURPY3VZYILWN6LY2X7COKLFDZKMEO6WXS4XA, seg_pos={0 0}, upload=461738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 27 13:53:40.446077987 (526h24m40.317161022s) [noise=true , address=45.91.116.4:4937 , piece_id=IEHFN6JGIQW34G7IURPY3VZYILWN6LY2X7COKLFDZKMEO6WXS4XA, seg_pos={0 0}, upload=461738]: Long tail cancellation due to optimal threshold Dec 27 13:53:40.446133633 (526h24m40.317216668s) [noise=true , address=45.91.116.8:4046 , piece_id=TLE7NOEIS747XZ3Z724GBISGZHBIER3JQRY66DPID55XVQ57ZU5Q, seg_pos={0 0}, upload=461738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:47622->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:47622->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47622->45.91.116.8:4046: use of closed network connection Dec 27 13:53:40.446138813 (526h24m40.317221848s) [noise=true , address=45.91.116.8:4046 , piece_id=TLE7NOEIS747XZ3Z724GBISGZHBIER3JQRY66DPID55XVQ57ZU5Q, seg_pos={0 0}, upload=461738]: Long tail cancellation due to optimal threshold Dec 27 13:53:40.446153611 (526h24m40.317236656s) [seg_pos={0 0}, upload=461738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:53:40.446228083 (526h24m40.317311118s) [upload=461738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:53:40.446232631 (526h24m40.317315666s) [upload=461738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:53:40.739780236 (526h24m40.610863281s) [upload=461738]: Done uploading segments... Dec 27 13:53:40.739796758 (526h24m40.610879793s) [upload=461738]: Upload finished. err: <nil> Duration: 5.880794193s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [.......XXXXX..XXXXX....XX....X.X.X.X.X..XXX.XX.XX.X.XX.X.XXX] duration: 5.001367065s trace: https://jaeger.dev.storj.io/trace/785df6b44e8fa471 FAIL (2025-12-27 13:54:09 UTC) orbiter warning: context deadline exceeded Duration: 5.001389035s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [...X..X...X.XXX....X...X...X...XX.......X....X..XX.X.XXXX..X] duration: 30.002607393s trace: https://jaeger.dev.storj.io/trace/6a91a73558fd48b8 FAIL (2025-12-27 13:54:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== egments... Dec 27 13:53:37.731749182 (526h24m37.602832217s) [upload=461738]: Got next segment. Inline: false Dec 27 13:53:37.731797104 (526h24m37.602880139s) [upload=461738]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:53:37.731801142 (526h24m37.602884177s) [upload=461738]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:53:37.979178972 (526h24m37.850262007s) [seg_pos={0 0}, upload=461738]: Begin upload segment... Dec 27 13:53:37.979241551 (526h24m37.850324586s) [seg_pos={0 0}, upload=461738]: Done begin upload segment. Dec 27 13:53:37.979264555 (526h24m37.8503476s) [noise=true , address=145.79.227.4:4187 , piece_id=6ZP3H5LSTA277NHYNRKHVUI3GXR53DSCYJDH2FXVPLQ46BDJTXCQ, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:37.979281617 (526h24m37.850364663s) [noise=true , address=45.91.116.5:4298 , piece_id=4YA3J7U3UKHSPDO2I44AIQ22RVPISYUIDFQVH56FANXNWV27SGAQ, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:37.979451561 (526h24m37.850534606s) [noise=true , address=145.79.227.29:4943 , piece_id=42SBBOV57XGWGT4CE4X4YXZGLGH3JAJ5K7CRF5QQ3ZWMLVNFZCGA, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:37.979907429 (526h24m37.850990474s) [noise=true , address=45.91.116.14:4189 , piece_id=FSN3WGFV6MVAPS64SH72DW3HQLLA5DWDSJK4LMEF6VYR62YOCMQQ, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:37.980327860 (526h24m37.851410905s) [noise=true , address=45.91.116.4:4937 , piece_id=IEHFN6JGIQW34G7IURPY3VZYILWN6LY2X7COKLFDZKMEO6WXS4XA, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:37.980561855 (526h24m37.85164489s) [noise=true , address=45.91.116.6:4973 , piece_id=JHXQTGS5ZK4HFNHCQMCOT4ZWEIDVROIZK7B5RJ5VVHCGNTE6MMRQ, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:37.981010670 (526h24m37.852093715s) [noise=true , address=145.79.227.45:4231 , piece_id=2NT4YO5R5WR2SW6XGCVQJ3CTSMPSLA64QW4PAHZ72ZHK5HEU6SPQ, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:37.981293538 (526h24m37.852376573s) [noise=true , address=45.91.116.8:4046 , piece_id=TLE7NOEIS747XZ3Z724GBISGZHBIER3JQRY66DPID55XVQ57ZU5Q, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:37.981567590 (526h24m37.852650625s) [noise=true , address=45.91.116.12:4226 , piece_id=GRNEZRU2ST5NLP5B37GEVBKZF76YNJYI4BV2IOHIXATWTQ2F2Q2A, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:37.981879364 (526h24m37.852962399s) [noise=true , address=45.91.116.10:4082 , piece_id=53FIQMUADW73MC7T5BO4QFBEXAYO4FB5KPYE65QE2BASJZF6B2LQ, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:37.982273143 (526h24m37.853356178s) [noise=true , address=145.79.227.17:4972 , piece_id=DEI2KTZI2AADI2BTB3SUTM24PEY44HCAURX7RBBXU7VZFK4EMS4A, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:37.982648629 (526h24m37.853731664s) [noise=true , address=145.79.227.47:4265 , piece_id=3IU4FJTXFR33UF6EQKDKQ3TMV7BL4CXMC6QZYB5RQMWEWLNKRQNA, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:37.983089719 (526h24m37.854172764s) [noise=true , address=145.79.227.39:4014 , piece_id=O27D7BJLIMTATT2CPPBOWVZAY7Z4XM5J7QNJBEFOZX5VIPOKEI5Q, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:37.983419777 (526h24m37.854502822s) [noise=true , address=145.79.227.26:4655 , piece_id=YSPCEP4SNAHGXLUS6EJWEAFH7RFGUZQ75BD62FBHDPNYNG5N34JA, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:37.983808368 (526h24m37.854891403s) [noise=true , address=145.79.227.42:4122 , piece_id=UWCCJ6OSPQUX435UEYQ7FSN5PLWY74Y6DQM2ZE6BUGGBXUJKH6WA, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:37.985006178 (526h24m37.856089213s) [noise=true , address=145.79.227.52:4446 , piece_id=YX4RYQ64OMLM5OYGIR62TILRCL2JP2PFDW275SK3D2X24EL7HGAA, seg_pos={0 0}, upload=461738]: Uploading piece... Dec 27 13:53:39.607227113 (526h24m39.478310148s) [upload=461738]: Next returned nil segment Dec 27 13:53:39.607242942 (526h24m39.478325987s) [upload=461738]: Waiting for error group managing segments... Dec 27 13:53:39.979288701 (526h24m39.850371736s) [noise=true , address=145.79.227.26:4655 , piece_id=YSPCEP4SNAHGXLUS6EJWEAFH7RFGUZQ75BD62FBHDPNYNG5N34JA, seg_pos={0 0}, upload=461738]: Done uploading piece. err: <nil> Dec 27 13:53:39.991948717 (526h24m39.863031752s) [noise=true , address=145.79.227.42:4122 , piece_id=UWCCJ6OSPQUX435UEYQ7FSN5PLWY74Y6DQM2ZE6BUGGBXUJKH6WA, seg_pos={0 0}, upload=461738]: Done uploading piece. err: <nil> Dec 27 13:53:39.992192942 (526h24m39.863275977s) [noise=true , address=145.79.227.39:4014 , piece_id=O27D7BJLIMTATT2CPPBOWVZAY7Z4XM5J7QNJBEFOZX5VIPOKEI5Q, seg_pos={0 0}, upload=461738]: Done uploading piece. err: <nil> Dec 27 13:53:39.995436820 (526h24m39.866519865s) [noise=true , address=145.79.227.45:4231 , piece_id=2NT4YO5R5WR2SW6XGCVQJ3CTSMPSLA64QW4PAHZ72ZHK5HEU6SPQ, seg_pos={0 0}, upload=461738]: Done uploading piece. err: <nil> Dec 27 13:53:39.995998078 (526h24m39.867081114s) [noise=true , address=145.79.227.17:4972 , piece_id=DEI2KTZI2AADI2BTB3SUTM24PEY44HCAURX7RBBXU7VZFK4EMS4A, seg_pos={0 0}, upload=461738]: Done uploading piece. err: <nil> Dec 27 13:53:40.001367613 (526h24m39.872450678s) [noise=true , address=145.79.227.47:4265 , piece_id=3IU4FJTXFR33UF6EQKDKQ3TMV7BL4CXMC6QZYB5RQMWEWLNKRQNA, seg_pos={0 0}, upload=461738]: Done uploading piece. err: <nil> Dec 27 13:53:40.001643679 (526h24m39.872726754s) [noise=true , address=145.79.227.4:4187 , piece_id=6ZP3H5LSTA277NHYNRKHVUI3GXR53DSCYJDH2FXVPLQ46BDJTXCQ, seg_pos={0 0}, upload=461738]: Done uploading piece. err: <nil> Dec 27 13:53:40.017537273 (526h24m39.888620308s) [noise=true , address=145.79.227.29:4943 , piece_id=42SBBOV57XGWGT4CE4X4YXZGLGH3JAJ5K7CRF5QQ3ZWMLVNFZCGA, seg_pos={0 0}, upload=461738]: Done uploading piece. err: <nil> Dec 27 13:53:40.019145205 (526h24m39.89022825s) [noise=true , address=145.79.227.52:4446 , piece_id=YX4RYQ64OMLM5OYGIR62TILRCL2JP2PFDW275SK3D2X24EL7HGAA, seg_pos={0 0}, upload=461738]: Done uploading piece. err: <nil> Dec 27 13:53:40.247411182 (526h24m40.118494227s) [noise=true , address=45.91.116.5:4298 , piece_id=4YA3J7U3UKHSPDO2I44AIQ22RVPISYUIDFQVH56FANXNWV27SGAQ, seg_pos={0 0}, upload=461738]: Done uploading piece. err: <nil> Dec 27 13:53:40.265252304 (526h24m40.136335339s) [noise=true , address=45.91.116.14:4189 , piece_id=FSN3WGFV6MVAPS64SH72DW3HQLLA5DWDSJK4LMEF6VYR62YOCMQQ, seg_pos={0 0}, upload=461738]: Done uploading piece. err: <nil> Dec 27 13:53:40.275178465 (526h24m40.14626151s) [noise=true , address=45.91.116.12:4226 , piece_id=GRNEZRU2ST5NLP5B37GEVBKZF76YNJYI4BV2IOHIXATWTQ2F2Q2A, seg_pos={0 0}, upload=461738]: Done uploading piece. err: <nil> Dec 27 13:53:40.445530043 (526h24m40.316613088s) [noise=true , address=45.91.116.6:4973 , piece_id=JHXQTGS5ZK4HFNHCQMCOT4ZWEIDVROIZK7B5RJ5VVHCGNTE6MMRQ, seg_pos={0 0}, upload=461738]: Done uploading piece. err: <nil> Dec 27 13:53:40.445550182 (526h24m40.316633227s) [seg_pos={0 0}, upload=461738]: Segment reached optimal threshold of 13 pieces. Dec 27 13:53:40.445786192 (526h24m40.316869227s) [noise=true , address=45.91.116.10:4082 , piece_id=53FIQMUADW73MC7T5BO4QFBEXAYO4FB5KPYE65QE2BASJZF6B2LQ, seg_pos={0 0}, upload=461738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:46986->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:46986->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46986->45.91.116.10:4082: use of closed network connection Dec 27 13:53:40.445793215 (526h24m40.31687625s) [noise=true , address=45.91.116.10:4082 , piece_id=53FIQMUADW73MC7T5BO4QFBEXAYO4FB5KPYE65QE2BASJZF6B2LQ, seg_pos={0 0}, upload=461738]: Long tail cancellation due to optimal threshold Dec 27 13:53:40.446068259 (526h24m40.317151294s) [noise=true , address=45.91.116.4:4937 , piece_id=IEHFN6JGIQW34G7IURPY3VZYILWN6LY2X7COKLFDZKMEO6WXS4XA, seg_pos={0 0}, upload=461738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 27 13:53:40.446077987 (526h24m40.317161022s) [noise=true , address=45.91.116.4:4937 , piece_id=IEHFN6JGIQW34G7IURPY3VZYILWN6LY2X7COKLFDZKMEO6WXS4XA, seg_pos={0 0}, upload=461738]: Long tail cancellation due to optimal threshold Dec 27 13:53:40.446133633 (526h24m40.317216668s) [noise=true , address=45.91.116.8:4046 , piece_id=TLE7NOEIS747XZ3Z724GBISGZHBIER3JQRY66DPID55XVQ57ZU5Q, seg_pos={0 0}, upload=461738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:47622->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:47622->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47622->45.91.116.8:4046: use of closed network connection Dec 27 13:53:40.446138813 (526h24m40.317221848s) [noise=true , address=45.91.116.8:4046 , piece_id=TLE7NOEIS747XZ3Z724GBISGZHBIER3JQRY66DPID55XVQ57ZU5Q, seg_pos={0 0}, upload=461738]: Long tail cancellation due to optimal threshold Dec 27 13:53:40.446153611 (526h24m40.317236656s) [seg_pos={0 0}, upload=461738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:53:40.446228083 (526h24m40.317311118s) [upload=461738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:53:40.446232631 (526h24m40.317315666s) [upload=461738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:53:40.739780236 (526h24m40.610863281s) [upload=461738]: Done uploading segments... Dec 27 13:53:40.739796758 (526h24m40.610879793s) [upload=461738]: Upload finished. err: <nil> Duration: 30.002625559s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............X...X...X..........X.............XX..........] duration: 59.173451892s trace: https://jaeger.dev.storj.io/trace/3ed1e657d506c4ee FAIL (2025-12-27 13:54:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, Piece ID: HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ) context deadline exceeded; piecestore: (Node ID: 12wXqePK1kK7iBo6wHcbqkRmkED8LGJ3rzQNJAWa14cHYhx1oyB, Piece ID: M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA) context deadline exceeded; piecestore: (Node ID: 12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx, Piece ID: ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ) context deadline exceeded; piecestore: (Node ID: 12UjFkFn2UEvr57LDorcijDCvkBkkKitgZEFx6Yqfk5gMStQvJG, Piece ID: SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ) context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.16:4739 , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.319646322 (526h14m34.190729357s) [noise=true , address=77.90.31.21:4899 , piece_id=SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.319933248 (526h14m34.191016283s) [noise=true , address=45.91.116.12:4234 , piece_id=Y2FIVYLZBQAYBO2TDGIZ4RDQHQJ3WOQH565A7ESIKANQQTQDFQZQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320400588 (526h14m34.191483633s) [noise=true , address=145.79.227.29:4952 , piece_id=VXZLF4OGRIR3MYQMMLKLTD235A55F4YTHB3D4YCVCQUL4FZTSFJQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320407341 (526h14m34.191490386s) [noise=true , address=145.79.227.46:4621 , piece_id=XZ4MKAZAGQKVEJJGSZ5QGT4AT3MFXXNMXICVFCI6SP3E4EVEHZGQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320675030 (526h14m34.191758075s) [noise=true , address=45.91.116.10:4113 , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320802022 (526h14m34.191885067s) [noise=true , address=77.90.31.11:4569 , piece_id=YC2Q3HPJDVY3NEDZNC2Z4KVN6S6DDDGRARASF7KALFAOQ335A7HA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.321169643 (526h14m34.192252678s) [noise=true , address=77.90.31.19:4869 , piece_id=HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.321443483 (526h14m34.192526518s) [noise=true , address=45.91.116.14:4202 , piece_id=M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.321731813 (526h14m34.192814848s) [noise=true , address=77.90.31.18:4819 , piece_id=YS4DXTHLGCPFOKKLBA2DK2BVOAPTQEGVOPBSRCLFV26SVP6YFHGA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.322221776 (526h14m34.193304811s) [noise=true , address=77.90.31.15:4708 , piece_id=3NX6E54XMZXQAQQE27LTWUSEF5GAIF2XOPQFFAP7LOA2PL7KDL3Q, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.322547776 (526h14m34.193630811s) [noise=true , address=145.79.227.40:4079 , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.322908093 (526h14m34.193991138s) [noise=true , address=145.79.227.29:4975 , piece_id=DAMNBVP7DRSW5OAGYRRP4K5SNJ2WEUT7IAXO77KWRHLQEMBNERZQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.323267968 (526h14m34.194351003s) [noise=true , address=145.79.227.30:4787 , piece_id=OB2OTY3ZEHIUFYM6FY7HIS7HQ353Y2HL6XU4476HXOD26QVA4DPQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.323618165 (526h14m34.19470121s) [noise=true , address=64.49.13.169:4710 , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.324084893 (526h14m34.195167928s) [noise=true , address=193.149.164.143:4694 , piece_id=ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:35.783540136 (526h14m35.654623181s) [upload=461595]: Next returned nil segment Dec 27 13:43:35.783555856 (526h14m35.654638891s) [upload=461595]: Waiting for error group managing segments... Dec 27 13:43:36.192287250 (526h14m36.063370296s) [noise=true , address=145.79.227.29:4952 , piece_id=VXZLF4OGRIR3MYQMMLKLTD235A55F4YTHB3D4YCVCQUL4FZTSFJQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.200200289 (526h14m36.071283334s) [noise=true , address=145.79.227.46:4621 , piece_id=XZ4MKAZAGQKVEJJGSZ5QGT4AT3MFXXNMXICVFCI6SP3E4EVEHZGQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.238087532 (526h14m36.109170577s) [noise=true , address=145.79.227.29:4975 , piece_id=DAMNBVP7DRSW5OAGYRRP4K5SNJ2WEUT7IAXO77KWRHLQEMBNERZQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.379631346 (526h14m36.250714391s) [noise=true , address=77.90.31.19:4869 , piece_id=HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.382945286 (526h14m36.254028321s) [noise=true , address=145.79.227.34:4909 , piece_id=Z6RF5PSOGOZMOXPQYXHMXGFXIELVXXVU4NXXRTAQCIELQXLMVXBQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.400290103 (526h14m36.271373138s) [noise=true , address=77.90.31.18:4819 , piece_id=YS4DXTHLGCPFOKKLBA2DK2BVOAPTQEGVOPBSRCLFV26SVP6YFHGA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.436707208 (526h14m36.307790243s) [noise=true , address=45.91.116.14:4202 , piece_id=M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.487651475 (526h14m36.35873451s) [noise=true , address=45.91.116.12:4234 , piece_id=Y2FIVYLZBQAYBO2TDGIZ4RDQHQJ3WOQH565A7ESIKANQQTQDFQZQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.500850769 (526h14m36.371933814s) [noise=true , address=77.90.31.11:4569 , piece_id=YC2Q3HPJDVY3NEDZNC2Z4KVN6S6DDDGRARASF7KALFAOQ335A7HA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.518828551 (526h14m36.389911586s) [noise=true , address=77.90.31.15:4708 , piece_id=3NX6E54XMZXQAQQE27LTWUSEF5GAIF2XOPQFFAP7LOA2PL7KDL3Q, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.051607243 (526h14m36.922690289s) [noise=true , address=145.79.227.44:4228 , piece_id=6SAYEAOTTCOUFPU75C7YRIULWJGXL4PK2YKCKFB47EUBWMRQQQOQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.096826909 (526h14m36.967909954s) [noise=true , address=145.79.227.30:4787 , piece_id=OB2OTY3ZEHIUFYM6FY7HIS7HQ353Y2HL6XU4476HXOD26QVA4DPQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.140738995 (526h14m37.01182203s) [noise=true , address=77.90.31.21:4899 , piece_id=SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.323613990 (526h14m37.194697035s) [noise=true , address=193.149.164.143:4694 , piece_id=ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.323649618 (526h14m37.194732663s) [seg_pos={0 0}, upload=461595]: Segment reached optimal threshold of 14 pieces. Dec 27 13:43:37.323852865 (526h14m37.1949359s) [noise=true , address=45.91.116.10:4113 , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12puRMny4nbgv2by4tAbXcBBSgG2d6BfwcDDRDcFzEdjSVUJxtd): protocol: expected piece hash; context canceled; EOF Dec 27 13:43:37.323860349 (526h14m37.194943384s) [noise=true , address=45.91.116.10:4113 , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324005135 (526h14m37.1950882s) [noise=true , address=145.79.227.40:4079 , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RYKzn5zVSmxVbWwnBdX8oqtG1WE2MhPeEiWLf5dZeFAut6s2Y): protocol: expected piece hash; context canceled; EOF Dec 27 13:43:37.324022659 (526h14m37.195105704s) [noise=true , address=145.79.227.40:4079 , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324143378 (526h14m37.195226413s) [noise=true , address=77.90.31.16:4739 , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129B3N7yjJvwCzoVLTqWsPbcLQ5TQUyAzg1RCkbSAVKPWYTG8eb): protocol: expected piece hash; context canceled; EOF Dec 27 13:43:37.324154520 (526h14m37.195237555s) [noise=true , address=77.90.31.16:4739 , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324213442 (526h14m37.195296487s) [noise=true , address=64.49.13.169:4710 , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection Dec 27 13:43:37.324249360 (526h14m37.195332405s) [noise=true , address=64.49.13.169:4710 , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324275360 (526h14m37.195358405s) [seg_pos={0 0}, upload=461595]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 13:43:37.324374018 (526h14m37.195457053s) [upload=461595]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:43:37.324381291 (526h14m37.195464337s) [upload=461595]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:43:37.679036406 (526h14m37.550119451s) [upload=461595]: Done uploading segments... Dec 27 13:43:37.679056985 (526h14m37.55014003s) [upload=461595]: Upload finished. err: <nil> Duration: 59.173485666s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [....X..X...X.XXX....X...X...X...XX.......X....X..XX.X.XXXX..] duration: 6.040638219s trace: https://jaeger.dev.storj.io/trace/6bd7086d3176aa3f FAIL (2025-12-27 13:53:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: 4PEGRMMP2WDUFDKJYT34UISE53T5ZGVVBQBS763SDQBU7W4R57AA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: WEC7XZMIW6JHMOLHGNCJVXTLXPGD3YGQFEBE5XNEL3JBFMEXQBUA) context deadline exceeded ======LOGS====== Dec 27 13:51:35.278044600 (526h22m35.149127635s) [upload=461710]: Starting upload Dec 27 13:51:35.278100016 (526h22m35.149183051s) [upload=461710]: Done starting upload Dec 27 13:51:35.278226776 (526h22m35.149309811s) [upload=461710]: Uploading segments... Dec 27 13:51:35.278381361 (526h22m35.149464426s) [upload=461710]: Got next segment. Inline: false Dec 27 13:51:35.278427609 (526h22m35.149510644s) [upload=461710]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:51:35.278434332 (526h22m35.149517367s) [upload=461710]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:51:35.524663684 (526h22m35.395746729s) [seg_pos={0 0}, upload=461710]: Begin upload segment... Dec 27 13:51:35.524724360 (526h22m35.395807395s) [seg_pos={0 0}, upload=461710]: Done begin upload segment. Dec 27 13:51:35.524776449 (526h22m35.395859484s) [noise=true , address=77.90.31.15:4700 , piece_id=O537VMBZHUKK7V6I6SPUTCX26K7CNGZK6ZU3AEKIRIDDO7EROIFQ, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:35.524829771 (526h22m35.395912806s) [noise=true , address=145.79.227.7:4294 , piece_id=ZVDV4F5WE4MBGJOGQZVDJ7U77G6ETVLYB5SS33M6R47C5ARAPSUA, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:35.525843421 (526h22m35.396926456s) [noise=true , address=145.79.227.2:4113 , piece_id=KJUVJ4OPWXESSJZ6UJS7PGYMHARDJIART5JRN5PH7SF3AQFEFMHQ, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:35.526098086 (526h22m35.397181121s) [noise=true , address=145.79.227.14:4837 , piece_id=B7TGZPTQNGP35ZUFWIW2PEQGRZVGS5IFBJKCM726HFE7BJXD7V3A, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:35.526514138 (526h22m35.397597173s) [noise=true , address=77.90.31.21:4880 , piece_id=4PEGRMMP2WDUFDKJYT34UISE53T5ZGVVBQBS763SDQBU7W4R57AA, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:35.526838717 (526h22m35.397921752s) [noise=true , address=77.90.31.14:4664 , piece_id=UXEUWGCPSFS5X7DYSY44V75KFNEY4KVIMCH2XSZ462Y6DNEEEIKQ, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:35.526902818 (526h22m35.397985853s) [noise=true , address=45.91.116.8:4048 , piece_id=WEC7XZMIW6JHMOLHGNCJVXTLXPGD3YGQFEBE5XNEL3JBFMEXQBUA, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:35.527241152 (526h22m35.398324197s) [noise=true , address=77.90.31.11:4556 , piece_id=WG53ZPGBZQGVGSF7LPH6NTOICQK3GIAQJP5QDWMAOHK3RE2CYKBA, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:35.527687001 (526h22m35.398770036s) [noise=true , address=45.91.116.6:4974 , piece_id=K6G74SMDZIR2FZJB2UHS6YFIKBJQDAYJLA3L5ZDFE5ICE6PGWZRA, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:35.527881191 (526h22m35.398964236s) [noise=true , address=145.79.227.5:4221 , piece_id=AZJTSPDXMY6HKWVDAD7PDR2I46AS26M4P2CGM356PXSE56FPI7AA, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:35.528004837 (526h22m35.399087872s) [noise=true , address=145.79.227.50:4373 , piece_id=BE6ALGZLEZK445IRVHI7BX3OXDHJ6QOODEX2P4ER4CZXWFGXOVTQ, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:35.528314426 (526h22m35.399397461s) [noise=true , address=145.79.227.51:4410 , piece_id=YFNJQPRTTZ626SQSMLVEKUU6MDGTFC3B5PDICPJWG625PKVBXN3Q, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:35.528512604 (526h22m35.399595639s) [noise=true , address=45.91.116.4:4940 , piece_id=PKRSPCMLL247WYWYDC547Q5KXFFA3M3CUI2VTXHA26DNY3XONXGA, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:35.528942422 (526h22m35.400025457s) [noise=true , address=77.90.31.18:4808 , piece_id=F2E2Y2NBQWU54DNMXF3FI7X36OHN7IQBVFD52UVQZGUU4JLVTAVQ, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:35.529576469 (526h22m35.400659504s) [noise=true , address=145.79.227.41:4085 , piece_id=LQLNZSQI37AY3CP6HWMS4HCY7GDKSB5KQK64VKNKPP45UGA5WV4A, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:35.529836364 (526h22m35.4009194s) [noise=true , address=145.79.227.12:4765 , piece_id=UVYQCLL3XUWLIFQPQ3PYYKMVWT5S36CFLKM5OMI27BA5DEXDBWTQ, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:37.245198126 (526h22m37.116281171s) [upload=461710]: Next returned nil segment Dec 27 13:51:37.245210559 (526h22m37.116293604s) [upload=461710]: Waiting for error group managing segments... Dec 27 13:51:37.531310251 (526h22m37.402393286s) [noise=true , address=145.79.227.14:4837 , piece_id=B7TGZPTQNGP35ZUFWIW2PEQGRZVGS5IFBJKCM726HFE7BJXD7V3A, seg_pos={0 0}, upload=461710]: Done uploading piece. err: <nil> Dec 27 13:51:37.531422997 (526h22m37.402506032s) [noise=true , address=145.79.227.7:4294 , piece_id=ZVDV4F5WE4MBGJOGQZVDJ7U77G6ETVLYB5SS33M6R47C5ARAPSUA, seg_pos={0 0}, upload=461710]: Done uploading piece. err: <nil> Dec 27 13:51:37.535571566 (526h22m37.406654601s) [noise=true , address=145.79.227.51:4410 , piece_id=YFNJQPRTTZ626SQSMLVEKUU6MDGTFC3B5PDICPJWG625PKVBXN3Q, seg_pos={0 0}, upload=461710]: Done uploading piece. err: <nil> Dec 27 13:51:37.540178949 (526h22m37.411261994s) [noise=true , address=145.79.227.12:4765 , piece_id=UVYQCLL3XUWLIFQPQ3PYYKMVWT5S36CFLKM5OMI27BA5DEXDBWTQ, seg_pos={0 0}, upload=461710]: Done uploading piece. err: <nil> Dec 27 13:51:37.550842345 (526h22m37.42192538s) [noise=true , address=145.79.227.2:4113 , piece_id=KJUVJ4OPWXESSJZ6UJS7PGYMHARDJIART5JRN5PH7SF3AQFEFMHQ, seg_pos={0 0}, upload=461710]: Done uploading piece. err: <nil> Dec 27 13:51:37.554248350 (526h22m37.425331395s) [noise=true , address=145.79.227.41:4085 , piece_id=LQLNZSQI37AY3CP6HWMS4HCY7GDKSB5KQK64VKNKPP45UGA5WV4A, seg_pos={0 0}, upload=461710]: Done uploading piece. err: <nil> Dec 27 13:51:37.554303295 (526h22m37.42538634s) [noise=true , address=145.79.227.50:4373 , piece_id=BE6ALGZLEZK445IRVHI7BX3OXDHJ6QOODEX2P4ER4CZXWFGXOVTQ, seg_pos={0 0}, upload=461710]: Done uploading piece. err: <nil> Dec 27 13:51:37.557086895 (526h22m37.42816993s) [noise=true , address=145.79.227.5:4221 , piece_id=AZJTSPDXMY6HKWVDAD7PDR2I46AS26M4P2CGM356PXSE56FPI7AA, seg_pos={0 0}, upload=461710]: Done uploading piece. err: <nil> Dec 27 13:51:37.803680942 (526h22m37.674763987s) [noise=true , address=45.91.116.8:4048 , piece_id=WEC7XZMIW6JHMOLHGNCJVXTLXPGD3YGQFEBE5XNEL3JBFMEXQBUA, seg_pos={0 0}, upload=461710]: Done uploading piece. err: <nil> Dec 27 13:51:37.824304543 (526h22m37.695387588s) [noise=true , address=77.90.31.11:4556 , piece_id=WG53ZPGBZQGVGSF7LPH6NTOICQK3GIAQJP5QDWMAOHK3RE2CYKBA, seg_pos={0 0}, upload=461710]: Done uploading piece. err: <nil> Dec 27 13:51:37.900976469 (526h22m37.772059514s) [noise=true , address=45.91.116.4:4940 , piece_id=PKRSPCMLL247WYWYDC547Q5KXFFA3M3CUI2VTXHA26DNY3XONXGA, seg_pos={0 0}, upload=461710]: Done uploading piece. err: <nil> Dec 27 13:51:38.427430698 (526h22m38.298513733s) [noise=true , address=77.90.31.21:4880 , piece_id=4PEGRMMP2WDUFDKJYT34UISE53T5ZGVVBQBS763SDQBU7W4R57AA, seg_pos={0 0}, upload=461710]: Done uploading piece. err: <nil> Dec 27 13:51:38.508863350 (526h22m38.379946395s) [noise=true , address=77.90.31.18:4808 , piece_id=F2E2Y2NBQWU54DNMXF3FI7X36OHN7IQBVFD52UVQZGUU4JLVTAVQ, seg_pos={0 0}, upload=461710]: Done uploading piece. err: <nil> Dec 27 13:51:38.508881755 (526h22m38.37996479s) [seg_pos={0 0}, upload=461710]: Segment reached optimal threshold of 13 pieces. Dec 27 13:51:38.509076726 (526h22m38.380159772s) [noise=true , address=45.91.116.6:4974 , piece_id=K6G74SMDZIR2FZJB2UHS6YFIKBJQDAYJLA3L5ZDFE5ICE6PGWZRA, seg_pos={0 0}, upload=461710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 27 13:51:38.509083479 (526h22m38.380166514s) [noise=true , address=45.91.116.6:4974 , piece_id=K6G74SMDZIR2FZJB2UHS6YFIKBJQDAYJLA3L5ZDFE5ICE6PGWZRA, seg_pos={0 0}, upload=461710]: Long tail cancellation due to optimal threshold Dec 27 13:51:38.509119618 (526h22m38.380202653s) [noise=true , address=77.90.31.14:4664 , piece_id=UXEUWGCPSFS5X7DYSY44V75KFNEY4KVIMCH2XSZ462Y6DNEEEIKQ, seg_pos={0 0}, upload=461710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 27 13:51:38.509127733 (526h22m38.380210768s) [noise=true , address=77.90.31.14:4664 , piece_id=UXEUWGCPSFS5X7DYSY44V75KFNEY4KVIMCH2XSZ462Y6DNEEEIKQ, seg_pos={0 0}, upload=461710]: Long tail cancellation due to optimal threshold Dec 27 13:51:38.509299661 (526h22m38.380382706s) [noise=true , address=77.90.31.15:4700 , piece_id=O537VMBZHUKK7V6I6SPUTCX26K7CNGZK6ZU3AEKIRIDDO7EROIFQ, seg_pos={0 0}, upload=461710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 27 13:51:38.509306303 (526h22m38.380389338s) [noise=true , address=77.90.31.15:4700 , piece_id=O537VMBZHUKK7V6I6SPUTCX26K7CNGZK6ZU3AEKIRIDDO7EROIFQ, seg_pos={0 0}, upload=461710]: Long tail cancellation due to optimal threshold Dec 27 13:51:38.509322915 (526h22m38.38040595s) [seg_pos={0 0}, upload=461710]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:51:38.509388410 (526h22m38.380471455s) [upload=461710]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:51:38.509392558 (526h22m38.380475603s) [upload=461710]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:51:38.804363993 (526h22m38.675447038s) [upload=461710]: Done uploading segments... Dec 27 13:51:38.804396405 (526h22m38.67547944s) [upload=461710]: Upload finished. err: <nil> Duration: 6.040666053s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [X.......XXXXX..XXXXX....XX....X.X.X.X.X..XXX.XX.XX.X.XX.X.XX] duration: 5.000800009s trace: https://jaeger.dev.storj.io/trace/f5e9148628c76bf FAIL (2025-12-27 13:53:20 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000826139s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [................X...X...X..........X.............XX.........] duration: 52.525786299s trace: https://jaeger.dev.storj.io/trace/3ed1e657d506c4ee FAIL (2025-12-27 13:52:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, Piece ID: HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ) context deadline exceeded; piecestore: (Node ID: 12wXqePK1kK7iBo6wHcbqkRmkED8LGJ3rzQNJAWa14cHYhx1oyB, Piece ID: M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA) context deadline exceeded; piecestore: (Node ID: 12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx, Piece ID: ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ) context deadline exceeded; piecestore: (Node ID: 12UjFkFn2UEvr57LDorcijDCvkBkkKitgZEFx6Yqfk5gMStQvJG, Piece ID: SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ) context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.16:4739 , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.319646322 (526h14m34.190729357s) [noise=true , address=77.90.31.21:4899 , piece_id=SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.319933248 (526h14m34.191016283s) [noise=true , address=45.91.116.12:4234 , piece_id=Y2FIVYLZBQAYBO2TDGIZ4RDQHQJ3WOQH565A7ESIKANQQTQDFQZQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320400588 (526h14m34.191483633s) [noise=true , address=145.79.227.29:4952 , piece_id=VXZLF4OGRIR3MYQMMLKLTD235A55F4YTHB3D4YCVCQUL4FZTSFJQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320407341 (526h14m34.191490386s) [noise=true , address=145.79.227.46:4621 , piece_id=XZ4MKAZAGQKVEJJGSZ5QGT4AT3MFXXNMXICVFCI6SP3E4EVEHZGQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320675030 (526h14m34.191758075s) [noise=true , address=45.91.116.10:4113 , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320802022 (526h14m34.191885067s) [noise=true , address=77.90.31.11:4569 , piece_id=YC2Q3HPJDVY3NEDZNC2Z4KVN6S6DDDGRARASF7KALFAOQ335A7HA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.321169643 (526h14m34.192252678s) [noise=true , address=77.90.31.19:4869 , piece_id=HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.321443483 (526h14m34.192526518s) [noise=true , address=45.91.116.14:4202 , piece_id=M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.321731813 (526h14m34.192814848s) [noise=true , address=77.90.31.18:4819 , piece_id=YS4DXTHLGCPFOKKLBA2DK2BVOAPTQEGVOPBSRCLFV26SVP6YFHGA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.322221776 (526h14m34.193304811s) [noise=true , address=77.90.31.15:4708 , piece_id=3NX6E54XMZXQAQQE27LTWUSEF5GAIF2XOPQFFAP7LOA2PL7KDL3Q, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.322547776 (526h14m34.193630811s) [noise=true , address=145.79.227.40:4079 , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.322908093 (526h14m34.193991138s) [noise=true , address=145.79.227.29:4975 , piece_id=DAMNBVP7DRSW5OAGYRRP4K5SNJ2WEUT7IAXO77KWRHLQEMBNERZQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.323267968 (526h14m34.194351003s) [noise=true , address=145.79.227.30:4787 , piece_id=OB2OTY3ZEHIUFYM6FY7HIS7HQ353Y2HL6XU4476HXOD26QVA4DPQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.323618165 (526h14m34.19470121s) [noise=true , address=64.49.13.169:4710 , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.324084893 (526h14m34.195167928s) [noise=true , address=193.149.164.143:4694 , piece_id=ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:35.783540136 (526h14m35.654623181s) [upload=461595]: Next returned nil segment Dec 27 13:43:35.783555856 (526h14m35.654638891s) [upload=461595]: Waiting for error group managing segments... Dec 27 13:43:36.192287250 (526h14m36.063370296s) [noise=true , address=145.79.227.29:4952 , piece_id=VXZLF4OGRIR3MYQMMLKLTD235A55F4YTHB3D4YCVCQUL4FZTSFJQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.200200289 (526h14m36.071283334s) [noise=true , address=145.79.227.46:4621 , piece_id=XZ4MKAZAGQKVEJJGSZ5QGT4AT3MFXXNMXICVFCI6SP3E4EVEHZGQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.238087532 (526h14m36.109170577s) [noise=true , address=145.79.227.29:4975 , piece_id=DAMNBVP7DRSW5OAGYRRP4K5SNJ2WEUT7IAXO77KWRHLQEMBNERZQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.379631346 (526h14m36.250714391s) [noise=true , address=77.90.31.19:4869 , piece_id=HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.382945286 (526h14m36.254028321s) [noise=true , address=145.79.227.34:4909 , piece_id=Z6RF5PSOGOZMOXPQYXHMXGFXIELVXXVU4NXXRTAQCIELQXLMVXBQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.400290103 (526h14m36.271373138s) [noise=true , address=77.90.31.18:4819 , piece_id=YS4DXTHLGCPFOKKLBA2DK2BVOAPTQEGVOPBSRCLFV26SVP6YFHGA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.436707208 (526h14m36.307790243s) [noise=true , address=45.91.116.14:4202 , piece_id=M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.487651475 (526h14m36.35873451s) [noise=true , address=45.91.116.12:4234 , piece_id=Y2FIVYLZBQAYBO2TDGIZ4RDQHQJ3WOQH565A7ESIKANQQTQDFQZQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.500850769 (526h14m36.371933814s) [noise=true , address=77.90.31.11:4569 , piece_id=YC2Q3HPJDVY3NEDZNC2Z4KVN6S6DDDGRARASF7KALFAOQ335A7HA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.518828551 (526h14m36.389911586s) [noise=true , address=77.90.31.15:4708 , piece_id=3NX6E54XMZXQAQQE27LTWUSEF5GAIF2XOPQFFAP7LOA2PL7KDL3Q, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.051607243 (526h14m36.922690289s) [noise=true , address=145.79.227.44:4228 , piece_id=6SAYEAOTTCOUFPU75C7YRIULWJGXL4PK2YKCKFB47EUBWMRQQQOQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.096826909 (526h14m36.967909954s) [noise=true , address=145.79.227.30:4787 , piece_id=OB2OTY3ZEHIUFYM6FY7HIS7HQ353Y2HL6XU4476HXOD26QVA4DPQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.140738995 (526h14m37.01182203s) [noise=true , address=77.90.31.21:4899 , piece_id=SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.323613990 (526h14m37.194697035s) [noise=true , address=193.149.164.143:4694 , piece_id=ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.323649618 (526h14m37.194732663s) [seg_pos={0 0}, upload=461595]: Segment reached optimal threshold of 14 pieces. Dec 27 13:43:37.323852865 (526h14m37.1949359s) [noise=true , address=45.91.116.10:4113 , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12puRMny4nbgv2by4tAbXcBBSgG2d6BfwcDDRDcFzEdjSVUJxtd): protocol: expected piece hash; context canceled; EOF Dec 27 13:43:37.323860349 (526h14m37.194943384s) [noise=true , address=45.91.116.10:4113 , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324005135 (526h14m37.1950882s) [noise=true , address=145.79.227.40:4079 , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RYKzn5zVSmxVbWwnBdX8oqtG1WE2MhPeEiWLf5dZeFAut6s2Y): protocol: expected piece hash; context canceled; EOF Dec 27 13:43:37.324022659 (526h14m37.195105704s) [noise=true , address=145.79.227.40:4079 , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324143378 (526h14m37.195226413s) [noise=true , address=77.90.31.16:4739 , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129B3N7yjJvwCzoVLTqWsPbcLQ5TQUyAzg1RCkbSAVKPWYTG8eb): protocol: expected piece hash; context canceled; EOF Dec 27 13:43:37.324154520 (526h14m37.195237555s) [noise=true , address=77.90.31.16:4739 , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324213442 (526h14m37.195296487s) [noise=true , address=64.49.13.169:4710 , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection Dec 27 13:43:37.324249360 (526h14m37.195332405s) [noise=true , address=64.49.13.169:4710 , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324275360 (526h14m37.195358405s) [seg_pos={0 0}, upload=461595]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 13:43:37.324374018 (526h14m37.195457053s) [upload=461595]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:43:37.324381291 (526h14m37.195464337s) [upload=461595]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:43:37.679036406 (526h14m37.550119451s) [upload=461595]: Done uploading segments... Dec 27 13:43:37.679056985 (526h14m37.55014003s) [upload=461595]: Upload finished. err: <nil> Duration: 52.525805425s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [.X.......XXXXX..XXXXX....XX....X.X.X.X.X..XXX.XX.XX.X.XX.X.X] duration: 5.001063994s trace: https://jaeger.dev.storj.io/trace/2e06b3884ee4fc99 FAIL (2025-12-27 13:52:32 UTC) orbiter warning: context deadline exceeded Duration: 5.001092419s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.....X..X...X.XXX....X...X...X...XX.......X....X..XX.X.XXXX.] duration: 5.893894064s trace: https://jaeger.dev.storj.io/trace/6bd7086d3176aa3f FAIL (2025-12-27 13:52:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: 4PEGRMMP2WDUFDKJYT34UISE53T5ZGVVBQBS763SDQBU7W4R57AA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: WEC7XZMIW6JHMOLHGNCJVXTLXPGD3YGQFEBE5XNEL3JBFMEXQBUA) context deadline exceeded ======LOGS====== Dec 27 13:51:35.278044600 (526h22m35.149127635s) [upload=461710]: Starting upload Dec 27 13:51:35.278100016 (526h22m35.149183051s) [upload=461710]: Done starting upload Dec 27 13:51:35.278226776 (526h22m35.149309811s) [upload=461710]: Uploading segments... Dec 27 13:51:35.278381361 (526h22m35.149464426s) [upload=461710]: Got next segment. Inline: false Dec 27 13:51:35.278427609 (526h22m35.149510644s) [upload=461710]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:51:35.278434332 (526h22m35.149517367s) [upload=461710]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:51:35.524663684 (526h22m35.395746729s) [seg_pos={0 0}, upload=461710]: Begin upload segment... Dec 27 13:51:35.524724360 (526h22m35.395807395s) [seg_pos={0 0}, upload=461710]: Done begin upload segment. Dec 27 13:51:35.524776449 (526h22m35.395859484s) [noise=true , address=77.90.31.15:4700 , piece_id=O537VMBZHUKK7V6I6SPUTCX26K7CNGZK6ZU3AEKIRIDDO7EROIFQ, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:35.524829771 (526h22m35.395912806s) [noise=true , address=145.79.227.7:4294 , piece_id=ZVDV4F5WE4MBGJOGQZVDJ7U77G6ETVLYB5SS33M6R47C5ARAPSUA, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:35.525843421 (526h22m35.396926456s) [noise=true , address=145.79.227.2:4113 , piece_id=KJUVJ4OPWXESSJZ6UJS7PGYMHARDJIART5JRN5PH7SF3AQFEFMHQ, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:35.526098086 (526h22m35.397181121s) [noise=true , address=145.79.227.14:4837 , piece_id=B7TGZPTQNGP35ZUFWIW2PEQGRZVGS5IFBJKCM726HFE7BJXD7V3A, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:35.526514138 (526h22m35.397597173s) [noise=true , address=77.90.31.21:4880 , piece_id=4PEGRMMP2WDUFDKJYT34UISE53T5ZGVVBQBS763SDQBU7W4R57AA, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:35.526838717 (526h22m35.397921752s) [noise=true , address=77.90.31.14:4664 , piece_id=UXEUWGCPSFS5X7DYSY44V75KFNEY4KVIMCH2XSZ462Y6DNEEEIKQ, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:35.526902818 (526h22m35.397985853s) [noise=true , address=45.91.116.8:4048 , piece_id=WEC7XZMIW6JHMOLHGNCJVXTLXPGD3YGQFEBE5XNEL3JBFMEXQBUA, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:35.527241152 (526h22m35.398324197s) [noise=true , address=77.90.31.11:4556 , piece_id=WG53ZPGBZQGVGSF7LPH6NTOICQK3GIAQJP5QDWMAOHK3RE2CYKBA, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:35.527687001 (526h22m35.398770036s) [noise=true , address=45.91.116.6:4974 , piece_id=K6G74SMDZIR2FZJB2UHS6YFIKBJQDAYJLA3L5ZDFE5ICE6PGWZRA, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:35.527881191 (526h22m35.398964236s) [noise=true , address=145.79.227.5:4221 , piece_id=AZJTSPDXMY6HKWVDAD7PDR2I46AS26M4P2CGM356PXSE56FPI7AA, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:35.528004837 (526h22m35.399087872s) [noise=true , address=145.79.227.50:4373 , piece_id=BE6ALGZLEZK445IRVHI7BX3OXDHJ6QOODEX2P4ER4CZXWFGXOVTQ, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:35.528314426 (526h22m35.399397461s) [noise=true , address=145.79.227.51:4410 , piece_id=YFNJQPRTTZ626SQSMLVEKUU6MDGTFC3B5PDICPJWG625PKVBXN3Q, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:35.528512604 (526h22m35.399595639s) [noise=true , address=45.91.116.4:4940 , piece_id=PKRSPCMLL247WYWYDC547Q5KXFFA3M3CUI2VTXHA26DNY3XONXGA, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:35.528942422 (526h22m35.400025457s) [noise=true , address=77.90.31.18:4808 , piece_id=F2E2Y2NBQWU54DNMXF3FI7X36OHN7IQBVFD52UVQZGUU4JLVTAVQ, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:35.529576469 (526h22m35.400659504s) [noise=true , address=145.79.227.41:4085 , piece_id=LQLNZSQI37AY3CP6HWMS4HCY7GDKSB5KQK64VKNKPP45UGA5WV4A, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:35.529836364 (526h22m35.4009194s) [noise=true , address=145.79.227.12:4765 , piece_id=UVYQCLL3XUWLIFQPQ3PYYKMVWT5S36CFLKM5OMI27BA5DEXDBWTQ, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:37.245198126 (526h22m37.116281171s) [upload=461710]: Next returned nil segment Dec 27 13:51:37.245210559 (526h22m37.116293604s) [upload=461710]: Waiting for error group managing segments... Dec 27 13:51:37.531310251 (526h22m37.402393286s) [noise=true , address=145.79.227.14:4837 , piece_id=B7TGZPTQNGP35ZUFWIW2PEQGRZVGS5IFBJKCM726HFE7BJXD7V3A, seg_pos={0 0}, upload=461710]: Done uploading piece. err: <nil> Dec 27 13:51:37.531422997 (526h22m37.402506032s) [noise=true , address=145.79.227.7:4294 , piece_id=ZVDV4F5WE4MBGJOGQZVDJ7U77G6ETVLYB5SS33M6R47C5ARAPSUA, seg_pos={0 0}, upload=461710]: Done uploading piece. err: <nil> Dec 27 13:51:37.535571566 (526h22m37.406654601s) [noise=true , address=145.79.227.51:4410 , piece_id=YFNJQPRTTZ626SQSMLVEKUU6MDGTFC3B5PDICPJWG625PKVBXN3Q, seg_pos={0 0}, upload=461710]: Done uploading piece. err: <nil> Dec 27 13:51:37.540178949 (526h22m37.411261994s) [noise=true , address=145.79.227.12:4765 , piece_id=UVYQCLL3XUWLIFQPQ3PYYKMVWT5S36CFLKM5OMI27BA5DEXDBWTQ, seg_pos={0 0}, upload=461710]: Done uploading piece. err: <nil> Dec 27 13:51:37.550842345 (526h22m37.42192538s) [noise=true , address=145.79.227.2:4113 , piece_id=KJUVJ4OPWXESSJZ6UJS7PGYMHARDJIART5JRN5PH7SF3AQFEFMHQ, seg_pos={0 0}, upload=461710]: Done uploading piece. err: <nil> Dec 27 13:51:37.554248350 (526h22m37.425331395s) [noise=true , address=145.79.227.41:4085 , piece_id=LQLNZSQI37AY3CP6HWMS4HCY7GDKSB5KQK64VKNKPP45UGA5WV4A, seg_pos={0 0}, upload=461710]: Done uploading piece. err: <nil> Dec 27 13:51:37.554303295 (526h22m37.42538634s) [noise=true , address=145.79.227.50:4373 , piece_id=BE6ALGZLEZK445IRVHI7BX3OXDHJ6QOODEX2P4ER4CZXWFGXOVTQ, seg_pos={0 0}, upload=461710]: Done uploading piece. err: <nil> Dec 27 13:51:37.557086895 (526h22m37.42816993s) [noise=true , address=145.79.227.5:4221 , piece_id=AZJTSPDXMY6HKWVDAD7PDR2I46AS26M4P2CGM356PXSE56FPI7AA, seg_pos={0 0}, upload=461710]: Done uploading piece. err: <nil> Dec 27 13:51:37.803680942 (526h22m37.674763987s) [noise=true , address=45.91.116.8:4048 , piece_id=WEC7XZMIW6JHMOLHGNCJVXTLXPGD3YGQFEBE5XNEL3JBFMEXQBUA, seg_pos={0 0}, upload=461710]: Done uploading piece. err: <nil> Dec 27 13:51:37.824304543 (526h22m37.695387588s) [noise=true , address=77.90.31.11:4556 , piece_id=WG53ZPGBZQGVGSF7LPH6NTOICQK3GIAQJP5QDWMAOHK3RE2CYKBA, seg_pos={0 0}, upload=461710]: Done uploading piece. err: <nil> Dec 27 13:51:37.900976469 (526h22m37.772059514s) [noise=true , address=45.91.116.4:4940 , piece_id=PKRSPCMLL247WYWYDC547Q5KXFFA3M3CUI2VTXHA26DNY3XONXGA, seg_pos={0 0}, upload=461710]: Done uploading piece. err: <nil> Dec 27 13:51:38.427430698 (526h22m38.298513733s) [noise=true , address=77.90.31.21:4880 , piece_id=4PEGRMMP2WDUFDKJYT34UISE53T5ZGVVBQBS763SDQBU7W4R57AA, seg_pos={0 0}, upload=461710]: Done uploading piece. err: <nil> Dec 27 13:51:38.508863350 (526h22m38.379946395s) [noise=true , address=77.90.31.18:4808 , piece_id=F2E2Y2NBQWU54DNMXF3FI7X36OHN7IQBVFD52UVQZGUU4JLVTAVQ, seg_pos={0 0}, upload=461710]: Done uploading piece. err: <nil> Dec 27 13:51:38.508881755 (526h22m38.37996479s) [seg_pos={0 0}, upload=461710]: Segment reached optimal threshold of 13 pieces. Dec 27 13:51:38.509076726 (526h22m38.380159772s) [noise=true , address=45.91.116.6:4974 , piece_id=K6G74SMDZIR2FZJB2UHS6YFIKBJQDAYJLA3L5ZDFE5ICE6PGWZRA, seg_pos={0 0}, upload=461710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 27 13:51:38.509083479 (526h22m38.380166514s) [noise=true , address=45.91.116.6:4974 , piece_id=K6G74SMDZIR2FZJB2UHS6YFIKBJQDAYJLA3L5ZDFE5ICE6PGWZRA, seg_pos={0 0}, upload=461710]: Long tail cancellation due to optimal threshold Dec 27 13:51:38.509119618 (526h22m38.380202653s) [noise=true , address=77.90.31.14:4664 , piece_id=UXEUWGCPSFS5X7DYSY44V75KFNEY4KVIMCH2XSZ462Y6DNEEEIKQ, seg_pos={0 0}, upload=461710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 27 13:51:38.509127733 (526h22m38.380210768s) [noise=true , address=77.90.31.14:4664 , piece_id=UXEUWGCPSFS5X7DYSY44V75KFNEY4KVIMCH2XSZ462Y6DNEEEIKQ, seg_pos={0 0}, upload=461710]: Long tail cancellation due to optimal threshold Dec 27 13:51:38.509299661 (526h22m38.380382706s) [noise=true , address=77.90.31.15:4700 , piece_id=O537VMBZHUKK7V6I6SPUTCX26K7CNGZK6ZU3AEKIRIDDO7EROIFQ, seg_pos={0 0}, upload=461710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 27 13:51:38.509306303 (526h22m38.380389338s) [noise=true , address=77.90.31.15:4700 , piece_id=O537VMBZHUKK7V6I6SPUTCX26K7CNGZK6ZU3AEKIRIDDO7EROIFQ, seg_pos={0 0}, upload=461710]: Long tail cancellation due to optimal threshold Dec 27 13:51:38.509322915 (526h22m38.38040595s) [seg_pos={0 0}, upload=461710]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:51:38.509388410 (526h22m38.380471455s) [upload=461710]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:51:38.509392558 (526h22m38.380475603s) [upload=461710]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:51:38.804363993 (526h22m38.675447038s) [upload=461710]: Done uploading segments... Dec 27 13:51:38.804396405 (526h22m38.67547944s) [upload=461710]: Upload finished. err: <nil> Duration: 5.893913461s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [......X..X...X.XXX....X...X...X...XX.......X....X..XX.X.XXXX] duration: 30.004508668s trace: https://jaeger.dev.storj.io/trace/6bd7086d3176aa3f FAIL (2025-12-27 13:52:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: 4PEGRMMP2WDUFDKJYT34UISE53T5ZGVVBQBS763SDQBU7W4R57AA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: WEC7XZMIW6JHMOLHGNCJVXTLXPGD3YGQFEBE5XNEL3JBFMEXQBUA) context deadline exceeded ======LOGS====== Dec 27 13:51:35.278044600 (526h22m35.149127635s) [upload=461710]: Starting upload Dec 27 13:51:35.278100016 (526h22m35.149183051s) [upload=461710]: Done starting upload Dec 27 13:51:35.278226776 (526h22m35.149309811s) [upload=461710]: Uploading segments... Dec 27 13:51:35.278381361 (526h22m35.149464426s) [upload=461710]: Got next segment. Inline: false Dec 27 13:51:35.278427609 (526h22m35.149510644s) [upload=461710]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:51:35.278434332 (526h22m35.149517367s) [upload=461710]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:51:35.524663684 (526h22m35.395746729s) [seg_pos={0 0}, upload=461710]: Begin upload segment... Dec 27 13:51:35.524724360 (526h22m35.395807395s) [seg_pos={0 0}, upload=461710]: Done begin upload segment. Dec 27 13:51:35.524776449 (526h22m35.395859484s) [noise=true , address=77.90.31.15:4700 , piece_id=O537VMBZHUKK7V6I6SPUTCX26K7CNGZK6ZU3AEKIRIDDO7EROIFQ, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:35.524829771 (526h22m35.395912806s) [noise=true , address=145.79.227.7:4294 , piece_id=ZVDV4F5WE4MBGJOGQZVDJ7U77G6ETVLYB5SS33M6R47C5ARAPSUA, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:35.525843421 (526h22m35.396926456s) [noise=true , address=145.79.227.2:4113 , piece_id=KJUVJ4OPWXESSJZ6UJS7PGYMHARDJIART5JRN5PH7SF3AQFEFMHQ, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:35.526098086 (526h22m35.397181121s) [noise=true , address=145.79.227.14:4837 , piece_id=B7TGZPTQNGP35ZUFWIW2PEQGRZVGS5IFBJKCM726HFE7BJXD7V3A, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:35.526514138 (526h22m35.397597173s) [noise=true , address=77.90.31.21:4880 , piece_id=4PEGRMMP2WDUFDKJYT34UISE53T5ZGVVBQBS763SDQBU7W4R57AA, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:35.526838717 (526h22m35.397921752s) [noise=true , address=77.90.31.14:4664 , piece_id=UXEUWGCPSFS5X7DYSY44V75KFNEY4KVIMCH2XSZ462Y6DNEEEIKQ, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:35.526902818 (526h22m35.397985853s) [noise=true , address=45.91.116.8:4048 , piece_id=WEC7XZMIW6JHMOLHGNCJVXTLXPGD3YGQFEBE5XNEL3JBFMEXQBUA, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:35.527241152 (526h22m35.398324197s) [noise=true , address=77.90.31.11:4556 , piece_id=WG53ZPGBZQGVGSF7LPH6NTOICQK3GIAQJP5QDWMAOHK3RE2CYKBA, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:35.527687001 (526h22m35.398770036s) [noise=true , address=45.91.116.6:4974 , piece_id=K6G74SMDZIR2FZJB2UHS6YFIKBJQDAYJLA3L5ZDFE5ICE6PGWZRA, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:35.527881191 (526h22m35.398964236s) [noise=true , address=145.79.227.5:4221 , piece_id=AZJTSPDXMY6HKWVDAD7PDR2I46AS26M4P2CGM356PXSE56FPI7AA, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:35.528004837 (526h22m35.399087872s) [noise=true , address=145.79.227.50:4373 , piece_id=BE6ALGZLEZK445IRVHI7BX3OXDHJ6QOODEX2P4ER4CZXWFGXOVTQ, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:35.528314426 (526h22m35.399397461s) [noise=true , address=145.79.227.51:4410 , piece_id=YFNJQPRTTZ626SQSMLVEKUU6MDGTFC3B5PDICPJWG625PKVBXN3Q, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:35.528512604 (526h22m35.399595639s) [noise=true , address=45.91.116.4:4940 , piece_id=PKRSPCMLL247WYWYDC547Q5KXFFA3M3CUI2VTXHA26DNY3XONXGA, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:35.528942422 (526h22m35.400025457s) [noise=true , address=77.90.31.18:4808 , piece_id=F2E2Y2NBQWU54DNMXF3FI7X36OHN7IQBVFD52UVQZGUU4JLVTAVQ, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:35.529576469 (526h22m35.400659504s) [noise=true , address=145.79.227.41:4085 , piece_id=LQLNZSQI37AY3CP6HWMS4HCY7GDKSB5KQK64VKNKPP45UGA5WV4A, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:35.529836364 (526h22m35.4009194s) [noise=true , address=145.79.227.12:4765 , piece_id=UVYQCLL3XUWLIFQPQ3PYYKMVWT5S36CFLKM5OMI27BA5DEXDBWTQ, seg_pos={0 0}, upload=461710]: Uploading piece... Dec 27 13:51:37.245198126 (526h22m37.116281171s) [upload=461710]: Next returned nil segment Dec 27 13:51:37.245210559 (526h22m37.116293604s) [upload=461710]: Waiting for error group managing segments... Dec 27 13:51:37.531310251 (526h22m37.402393286s) [noise=true , address=145.79.227.14:4837 , piece_id=B7TGZPTQNGP35ZUFWIW2PEQGRZVGS5IFBJKCM726HFE7BJXD7V3A, seg_pos={0 0}, upload=461710]: Done uploading piece. err: <nil> Dec 27 13:51:37.531422997 (526h22m37.402506032s) [noise=true , address=145.79.227.7:4294 , piece_id=ZVDV4F5WE4MBGJOGQZVDJ7U77G6ETVLYB5SS33M6R47C5ARAPSUA, seg_pos={0 0}, upload=461710]: Done uploading piece. err: <nil> Dec 27 13:51:37.535571566 (526h22m37.406654601s) [noise=true , address=145.79.227.51:4410 , piece_id=YFNJQPRTTZ626SQSMLVEKUU6MDGTFC3B5PDICPJWG625PKVBXN3Q, seg_pos={0 0}, upload=461710]: Done uploading piece. err: <nil> Dec 27 13:51:37.540178949 (526h22m37.411261994s) [noise=true , address=145.79.227.12:4765 , piece_id=UVYQCLL3XUWLIFQPQ3PYYKMVWT5S36CFLKM5OMI27BA5DEXDBWTQ, seg_pos={0 0}, upload=461710]: Done uploading piece. err: <nil> Dec 27 13:51:37.550842345 (526h22m37.42192538s) [noise=true , address=145.79.227.2:4113 , piece_id=KJUVJ4OPWXESSJZ6UJS7PGYMHARDJIART5JRN5PH7SF3AQFEFMHQ, seg_pos={0 0}, upload=461710]: Done uploading piece. err: <nil> Dec 27 13:51:37.554248350 (526h22m37.425331395s) [noise=true , address=145.79.227.41:4085 , piece_id=LQLNZSQI37AY3CP6HWMS4HCY7GDKSB5KQK64VKNKPP45UGA5WV4A, seg_pos={0 0}, upload=461710]: Done uploading piece. err: <nil> Dec 27 13:51:37.554303295 (526h22m37.42538634s) [noise=true , address=145.79.227.50:4373 , piece_id=BE6ALGZLEZK445IRVHI7BX3OXDHJ6QOODEX2P4ER4CZXWFGXOVTQ, seg_pos={0 0}, upload=461710]: Done uploading piece. err: <nil> Dec 27 13:51:37.557086895 (526h22m37.42816993s) [noise=true , address=145.79.227.5:4221 , piece_id=AZJTSPDXMY6HKWVDAD7PDR2I46AS26M4P2CGM356PXSE56FPI7AA, seg_pos={0 0}, upload=461710]: Done uploading piece. err: <nil> Dec 27 13:51:37.803680942 (526h22m37.674763987s) [noise=true , address=45.91.116.8:4048 , piece_id=WEC7XZMIW6JHMOLHGNCJVXTLXPGD3YGQFEBE5XNEL3JBFMEXQBUA, seg_pos={0 0}, upload=461710]: Done uploading piece. err: <nil> Dec 27 13:51:37.824304543 (526h22m37.695387588s) [noise=true , address=77.90.31.11:4556 , piece_id=WG53ZPGBZQGVGSF7LPH6NTOICQK3GIAQJP5QDWMAOHK3RE2CYKBA, seg_pos={0 0}, upload=461710]: Done uploading piece. err: <nil> Dec 27 13:51:37.900976469 (526h22m37.772059514s) [noise=true , address=45.91.116.4:4940 , piece_id=PKRSPCMLL247WYWYDC547Q5KXFFA3M3CUI2VTXHA26DNY3XONXGA, seg_pos={0 0}, upload=461710]: Done uploading piece. err: <nil> Dec 27 13:51:38.427430698 (526h22m38.298513733s) [noise=true , address=77.90.31.21:4880 , piece_id=4PEGRMMP2WDUFDKJYT34UISE53T5ZGVVBQBS763SDQBU7W4R57AA, seg_pos={0 0}, upload=461710]: Done uploading piece. err: <nil> Dec 27 13:51:38.508863350 (526h22m38.379946395s) [noise=true , address=77.90.31.18:4808 , piece_id=F2E2Y2NBQWU54DNMXF3FI7X36OHN7IQBVFD52UVQZGUU4JLVTAVQ, seg_pos={0 0}, upload=461710]: Done uploading piece. err: <nil> Dec 27 13:51:38.508881755 (526h22m38.37996479s) [seg_pos={0 0}, upload=461710]: Segment reached optimal threshold of 13 pieces. Dec 27 13:51:38.509076726 (526h22m38.380159772s) [noise=true , address=45.91.116.6:4974 , piece_id=K6G74SMDZIR2FZJB2UHS6YFIKBJQDAYJLA3L5ZDFE5ICE6PGWZRA, seg_pos={0 0}, upload=461710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 27 13:51:38.509083479 (526h22m38.380166514s) [noise=true , address=45.91.116.6:4974 , piece_id=K6G74SMDZIR2FZJB2UHS6YFIKBJQDAYJLA3L5ZDFE5ICE6PGWZRA, seg_pos={0 0}, upload=461710]: Long tail cancellation due to optimal threshold Dec 27 13:51:38.509119618 (526h22m38.380202653s) [noise=true , address=77.90.31.14:4664 , piece_id=UXEUWGCPSFS5X7DYSY44V75KFNEY4KVIMCH2XSZ462Y6DNEEEIKQ, seg_pos={0 0}, upload=461710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 27 13:51:38.509127733 (526h22m38.380210768s) [noise=true , address=77.90.31.14:4664 , piece_id=UXEUWGCPSFS5X7DYSY44V75KFNEY4KVIMCH2XSZ462Y6DNEEEIKQ, seg_pos={0 0}, upload=461710]: Long tail cancellation due to optimal threshold Dec 27 13:51:38.509299661 (526h22m38.380382706s) [noise=true , address=77.90.31.15:4700 , piece_id=O537VMBZHUKK7V6I6SPUTCX26K7CNGZK6ZU3AEKIRIDDO7EROIFQ, seg_pos={0 0}, upload=461710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 27 13:51:38.509306303 (526h22m38.380389338s) [noise=true , address=77.90.31.15:4700 , piece_id=O537VMBZHUKK7V6I6SPUTCX26K7CNGZK6ZU3AEKIRIDDO7EROIFQ, seg_pos={0 0}, upload=461710]: Long tail cancellation due to optimal threshold Dec 27 13:51:38.509322915 (526h22m38.38040595s) [seg_pos={0 0}, upload=461710]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:51:38.509388410 (526h22m38.380471455s) [upload=461710]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:51:38.509392558 (526h22m38.380475603s) [upload=461710]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:51:38.804363993 (526h22m38.675447038s) [upload=461710]: Done uploading segments... Dec 27 13:51:38.804396405 (526h22m38.67547944s) [upload=461710]: Upload finished. err: <nil> Duration: 30.004523997s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................X...X...X..........X.............XX........] duration: 47.47071859s trace: https://jaeger.dev.storj.io/trace/3ed1e657d506c4ee FAIL (2025-12-27 13:52:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, Piece ID: HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ) context deadline exceeded; piecestore: (Node ID: 12wXqePK1kK7iBo6wHcbqkRmkED8LGJ3rzQNJAWa14cHYhx1oyB, Piece ID: M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA) context deadline exceeded; piecestore: (Node ID: 12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx, Piece ID: ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ) context deadline exceeded; piecestore: (Node ID: 12UjFkFn2UEvr57LDorcijDCvkBkkKitgZEFx6Yqfk5gMStQvJG, Piece ID: SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ) context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.16:4739 , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.319646322 (526h14m34.190729357s) [noise=true , address=77.90.31.21:4899 , piece_id=SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.319933248 (526h14m34.191016283s) [noise=true , address=45.91.116.12:4234 , piece_id=Y2FIVYLZBQAYBO2TDGIZ4RDQHQJ3WOQH565A7ESIKANQQTQDFQZQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320400588 (526h14m34.191483633s) [noise=true , address=145.79.227.29:4952 , piece_id=VXZLF4OGRIR3MYQMMLKLTD235A55F4YTHB3D4YCVCQUL4FZTSFJQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320407341 (526h14m34.191490386s) [noise=true , address=145.79.227.46:4621 , piece_id=XZ4MKAZAGQKVEJJGSZ5QGT4AT3MFXXNMXICVFCI6SP3E4EVEHZGQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320675030 (526h14m34.191758075s) [noise=true , address=45.91.116.10:4113 , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320802022 (526h14m34.191885067s) [noise=true , address=77.90.31.11:4569 , piece_id=YC2Q3HPJDVY3NEDZNC2Z4KVN6S6DDDGRARASF7KALFAOQ335A7HA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.321169643 (526h14m34.192252678s) [noise=true , address=77.90.31.19:4869 , piece_id=HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.321443483 (526h14m34.192526518s) [noise=true , address=45.91.116.14:4202 , piece_id=M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.321731813 (526h14m34.192814848s) [noise=true , address=77.90.31.18:4819 , piece_id=YS4DXTHLGCPFOKKLBA2DK2BVOAPTQEGVOPBSRCLFV26SVP6YFHGA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.322221776 (526h14m34.193304811s) [noise=true , address=77.90.31.15:4708 , piece_id=3NX6E54XMZXQAQQE27LTWUSEF5GAIF2XOPQFFAP7LOA2PL7KDL3Q, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.322547776 (526h14m34.193630811s) [noise=true , address=145.79.227.40:4079 , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.322908093 (526h14m34.193991138s) [noise=true , address=145.79.227.29:4975 , piece_id=DAMNBVP7DRSW5OAGYRRP4K5SNJ2WEUT7IAXO77KWRHLQEMBNERZQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.323267968 (526h14m34.194351003s) [noise=true , address=145.79.227.30:4787 , piece_id=OB2OTY3ZEHIUFYM6FY7HIS7HQ353Y2HL6XU4476HXOD26QVA4DPQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.323618165 (526h14m34.19470121s) [noise=true , address=64.49.13.169:4710 , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.324084893 (526h14m34.195167928s) [noise=true , address=193.149.164.143:4694 , piece_id=ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:35.783540136 (526h14m35.654623181s) [upload=461595]: Next returned nil segment Dec 27 13:43:35.783555856 (526h14m35.654638891s) [upload=461595]: Waiting for error group managing segments... Dec 27 13:43:36.192287250 (526h14m36.063370296s) [noise=true , address=145.79.227.29:4952 , piece_id=VXZLF4OGRIR3MYQMMLKLTD235A55F4YTHB3D4YCVCQUL4FZTSFJQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.200200289 (526h14m36.071283334s) [noise=true , address=145.79.227.46:4621 , piece_id=XZ4MKAZAGQKVEJJGSZ5QGT4AT3MFXXNMXICVFCI6SP3E4EVEHZGQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.238087532 (526h14m36.109170577s) [noise=true , address=145.79.227.29:4975 , piece_id=DAMNBVP7DRSW5OAGYRRP4K5SNJ2WEUT7IAXO77KWRHLQEMBNERZQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.379631346 (526h14m36.250714391s) [noise=true , address=77.90.31.19:4869 , piece_id=HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.382945286 (526h14m36.254028321s) [noise=true , address=145.79.227.34:4909 , piece_id=Z6RF5PSOGOZMOXPQYXHMXGFXIELVXXVU4NXXRTAQCIELQXLMVXBQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.400290103 (526h14m36.271373138s) [noise=true , address=77.90.31.18:4819 , piece_id=YS4DXTHLGCPFOKKLBA2DK2BVOAPTQEGVOPBSRCLFV26SVP6YFHGA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.436707208 (526h14m36.307790243s) [noise=true , address=45.91.116.14:4202 , piece_id=M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.487651475 (526h14m36.35873451s) [noise=true , address=45.91.116.12:4234 , piece_id=Y2FIVYLZBQAYBO2TDGIZ4RDQHQJ3WOQH565A7ESIKANQQTQDFQZQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.500850769 (526h14m36.371933814s) [noise=true , address=77.90.31.11:4569 , piece_id=YC2Q3HPJDVY3NEDZNC2Z4KVN6S6DDDGRARASF7KALFAOQ335A7HA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.518828551 (526h14m36.389911586s) [noise=true , address=77.90.31.15:4708 , piece_id=3NX6E54XMZXQAQQE27LTWUSEF5GAIF2XOPQFFAP7LOA2PL7KDL3Q, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.051607243 (526h14m36.922690289s) [noise=true , address=145.79.227.44:4228 , piece_id=6SAYEAOTTCOUFPU75C7YRIULWJGXL4PK2YKCKFB47EUBWMRQQQOQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.096826909 (526h14m36.967909954s) [noise=true , address=145.79.227.30:4787 , piece_id=OB2OTY3ZEHIUFYM6FY7HIS7HQ353Y2HL6XU4476HXOD26QVA4DPQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.140738995 (526h14m37.01182203s) [noise=true , address=77.90.31.21:4899 , piece_id=SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.323613990 (526h14m37.194697035s) [noise=true , address=193.149.164.143:4694 , piece_id=ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.323649618 (526h14m37.194732663s) [seg_pos={0 0}, upload=461595]: Segment reached optimal threshold of 14 pieces. Dec 27 13:43:37.323852865 (526h14m37.1949359s) [noise=true , address=45.91.116.10:4113 , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12puRMny4nbgv2by4tAbXcBBSgG2d6BfwcDDRDcFzEdjSVUJxtd): protocol: expected piece hash; context canceled; EOF Dec 27 13:43:37.323860349 (526h14m37.194943384s) [noise=true , address=45.91.116.10:4113 , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324005135 (526h14m37.1950882s) [noise=true , address=145.79.227.40:4079 , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RYKzn5zVSmxVbWwnBdX8oqtG1WE2MhPeEiWLf5dZeFAut6s2Y): protocol: expected piece hash; context canceled; EOF Dec 27 13:43:37.324022659 (526h14m37.195105704s) [noise=true , address=145.79.227.40:4079 , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324143378 (526h14m37.195226413s) [noise=true , address=77.90.31.16:4739 , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129B3N7yjJvwCzoVLTqWsPbcLQ5TQUyAzg1RCkbSAVKPWYTG8eb): protocol: expected piece hash; context canceled; EOF Dec 27 13:43:37.324154520 (526h14m37.195237555s) [noise=true , address=77.90.31.16:4739 , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324213442 (526h14m37.195296487s) [noise=true , address=64.49.13.169:4710 , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection Dec 27 13:43:37.324249360 (526h14m37.195332405s) [noise=true , address=64.49.13.169:4710 , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324275360 (526h14m37.195358405s) [seg_pos={0 0}, upload=461595]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 13:43:37.324374018 (526h14m37.195457053s) [upload=461595]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:43:37.324381291 (526h14m37.195464337s) [upload=461595]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:43:37.679036406 (526h14m37.550119451s) [upload=461595]: Done uploading segments... Dec 27 13:43:37.679056985 (526h14m37.55014003s) [upload=461595]: Upload finished. err: <nil> Duration: 47.470738849s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [X.X.......XXXXX..XXXXX....XX....X.X.X.X.X..XXX.XX.XX.X.XX.X.] duration: 5.001088298s trace: https://jaeger.dev.storj.io/trace/450bc773be1f59c4 FAIL (2025-12-27 13:51:31 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001118376s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X......X..X...X.XXX....X...X...X...XX.......X....X..XX.X.XXX] duration: 30.002629941s trace: https://jaeger.dev.storj.io/trace/51f44d426e1611d2 FAIL (2025-12-27 13:51:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: 7CIWYC66SESQGY5RHN2RDABPKSC6YH7RVL2MPXKPBSYUTRK5ZXYQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: SM6W2CKCVET3QYOHUFJVIXQR5EDHIQ3RKFVVPT4WJLWL2SLHFZWQ) context deadline exceeded ======LOGS====== 13:50:55.559212404 (526h21m55.430295439s) [upload=461701]: Starting upload Dec 27 13:50:55.559269472 (526h21m55.430352507s) [upload=461701]: Done starting upload Dec 27 13:50:55.559400662 (526h21m55.430483697s) [upload=461701]: Uploading segments... Dec 27 13:50:55.559504710 (526h21m55.430587745s) [upload=461701]: Got next segment. Inline: false Dec 27 13:50:55.559545708 (526h21m55.430628743s) [upload=461701]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:50:55.559565256 (526h21m55.430648301s) [upload=461701]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:50:55.810211789 (526h21m55.681294824s) [seg_pos={0 0}, upload=461701]: Begin upload segment... Dec 27 13:50:55.810261613 (526h21m55.681344648s) [seg_pos={0 0}, upload=461701]: Done begin upload segment. Dec 27 13:50:55.810295798 (526h21m55.681378833s) [noise=true , address=145.79.227.26:4655 , piece_id=KNIGJ3QGFHUVQNHXQNVVCGSBAXT6UUD6EBSDES2ONOGRW6ABCF4Q, seg_pos={0 0}, upload=461701]: Uploading piece... Dec 27 13:50:55.810347207 (526h21m55.681430242s) [noise=true , address=45.91.116.4:4940 , piece_id=6NY4ART5MKDAIOMWIRZP3JZVHJYH56SZGJXOC4TUXQVT3QBV2H7A, seg_pos={0 0}, upload=461701]: Uploading piece... Dec 27 13:50:55.810789148 (526h21m55.681872183s) [noise=true , address=77.90.31.12:4590 , piece_id=POX7H3SSLNBQNY3VD62NXDYPSDFTBPGWDFZ2FECTVUCGZ3FVTTGA, seg_pos={0 0}, upload=461701]: Uploading piece... Dec 27 13:50:55.811100541 (526h21m55.682183586s) [noise=true , address=145.79.227.7:4294 , piece_id=KBZQZS6ANZMRC3STGIQVGXPROC4W3YPL7NYQO6XUPWZNVHNJ3UXA, seg_pos={0 0}, upload=461701]: Uploading piece... Dec 27 13:50:55.811509129 (526h21m55.682592164s) [noise=true , address=145.79.227.3:4150 , piece_id=4FK5BTSXTTYGCAXPSP7PB53DTW44FSUKYRZDVZBHQITZIOXOQABA, seg_pos={0 0}, upload=461701]: Uploading piece... Dec 27 13:50:55.811721703 (526h21m55.682804738s) [noise=true , address=145.79.227.45:4231 , piece_id=NRBUGNQMFDEALUJOWDXKHLMU5I2KDFMYO75BFEP7DJPWEN2CXPEQ, seg_pos={0 0}, upload=461701]: Uploading piece... Dec 27 13:50:55.811873663 (526h21m55.682956698s) [noise=true , address=145.79.227.32:4835 , piece_id=AJO6L64FFZOAII3IQ4QEFMWKH3PLG4USUNXLZH5HKPSULIZR2K3Q, seg_pos={0 0}, upload=461701]: Uploading piece... Dec 27 13:50:55.812156903 (526h21m55.683239938s) [noise=true , address=45.91.116.5:4298 , piece_id=7CIWYC66SESQGY5RHN2RDABPKSC6YH7RVL2MPXKPBSYUTRK5ZXYQ, seg_pos={0 0}, upload=461701]: Uploading piece... Dec 27 13:50:55.812190216 (526h21m55.683273261s) [noise=true , address=145.79.227.48:4302 , piece_id=O6UYF3LUNYIM47FQEFOGIYKINQS5Y2FOQGYS76346PSIN6YRRVMA, seg_pos={0 0}, upload=461701]: Uploading piece... Dec 27 13:50:55.812417358 (526h21m55.683500393s) [noise=true , address=145.79.227.16:4911 , piece_id=K6C545YF7FI646HFEPVBMQK3GIWMOZXNQDWDSNYMB7OYHDQPOSOA, seg_pos={0 0}, upload=461701]: Uploading piece... Dec 27 13:50:55.812701750 (526h21m55.683784795s) [noise=true , address=145.79.227.4:4187 , piece_id=PI2OUTI5AIJIH4DOENMMBSSKHBXNQZWJHTWOKO7ZHNVIIID4INZQ, seg_pos={0 0}, upload=461701]: Uploading piece... Dec 27 13:50:55.812730214 (526h21m55.683813249s) [noise=true , address=77.90.31.11:4554 , piece_id=BLMCQ5DBRGSXBGFCCXOU7YGLALSWSQIKWEVIVD5GNR6H4W52F2BQ, seg_pos={0 0}, upload=461701]: Uploading piece... Dec 27 13:50:55.812996140 (526h21m55.684079186s) [noise=true , address=145.79.227.19:4078 , piece_id=LW5GAX5IOH2PGBJ7Q3ZYRIEUYVWXM4TLKLXESXGADODCWSGB5OFA, seg_pos={0 0}, upload=461701]: Uploading piece... Dec 27 13:50:55.813133502 (526h21m55.684216547s) [noise=true , address=45.91.116.13:4153 , piece_id=SM6W2CKCVET3QYOHUFJVIXQR5EDHIQ3RKFVVPT4WJLWL2SLHFZWQ, seg_pos={0 0}, upload=461701]: Uploading piece... Dec 27 13:50:55.813606643 (526h21m55.684689678s) [noise=true , address=45.91.116.12:4226 , piece_id=HIOB42F74EPG5LXICFS53IMR734NCKC2AOTCYMD6ROITHDMLLRXA, seg_pos={0 0}, upload=461701]: Uploading piece... Dec 27 13:50:55.814172170 (526h21m55.685255215s) [noise=true , address=145.79.227.24:4583 , piece_id=7NWVAUIGVFUAXIPXA7TCP7VN6DBB75NQDRTALKF3QIAZT3Y47TLA, seg_pos={0 0}, upload=461701]: Uploading piece... Dec 27 13:50:57.520556098 (526h21m57.391639143s) [upload=461701]: Next returned nil segment Dec 27 13:50:57.520569103 (526h21m57.391652138s) [upload=461701]: Waiting for error group managing segments... Dec 27 13:50:57.803870247 (526h21m57.674953292s) [noise=true , address=145.79.227.7:4294 , piece_id=KBZQZS6ANZMRC3STGIQVGXPROC4W3YPL7NYQO6XUPWZNVHNJ3UXA, seg_pos={0 0}, upload=461701]: Done uploading piece. err: <nil> Dec 27 13:50:57.814640986 (526h21m57.685724021s) [noise=true , address=145.79.227.26:4655 , piece_id=KNIGJ3QGFHUVQNHXQNVVCGSBAXT6UUD6EBSDES2ONOGRW6ABCF4Q, seg_pos={0 0}, upload=461701]: Done uploading piece. err: <nil> Dec 27 13:50:57.817172457 (526h21m57.688255492s) [noise=true , address=145.79.227.32:4835 , piece_id=AJO6L64FFZOAII3IQ4QEFMWKH3PLG4USUNXLZH5HKPSULIZR2K3Q, seg_pos={0 0}, upload=461701]: Done uploading piece. err: <nil> Dec 27 13:50:57.818646483 (526h21m57.689729518s) [noise=true , address=145.79.227.45:4231 , piece_id=NRBUGNQMFDEALUJOWDXKHLMU5I2KDFMYO75BFEP7DJPWEN2CXPEQ, seg_pos={0 0}, upload=461701]: Done uploading piece. err: <nil> Dec 27 13:50:57.821531166 (526h21m57.692614211s) [noise=true , address=145.79.227.24:4583 , piece_id=7NWVAUIGVFUAXIPXA7TCP7VN6DBB75NQDRTALKF3QIAZT3Y47TLA, seg_pos={0 0}, upload=461701]: Done uploading piece. err: <nil> Dec 27 13:50:57.823109702 (526h21m57.694192737s) [noise=true , address=145.79.227.48:4302 , piece_id=O6UYF3LUNYIM47FQEFOGIYKINQS5Y2FOQGYS76346PSIN6YRRVMA, seg_pos={0 0}, upload=461701]: Done uploading piece. err: <nil> Dec 27 13:50:57.841370553 (526h21m57.712453588s) [noise=true , address=145.79.227.3:4150 , piece_id=4FK5BTSXTTYGCAXPSP7PB53DTW44FSUKYRZDVZBHQITZIOXOQABA, seg_pos={0 0}, upload=461701]: Done uploading piece. err: <nil> Dec 27 13:50:57.841430087 (526h21m57.712513132s) [noise=true , address=145.79.227.19:4078 , piece_id=LW5GAX5IOH2PGBJ7Q3ZYRIEUYVWXM4TLKLXESXGADODCWSGB5OFA, seg_pos={0 0}, upload=461701]: Done uploading piece. err: <nil> Dec 27 13:50:57.849108288 (526h21m57.720191323s) [noise=true , address=145.79.227.16:4911 , piece_id=K6C545YF7FI646HFEPVBMQK3GIWMOZXNQDWDSNYMB7OYHDQPOSOA, seg_pos={0 0}, upload=461701]: Done uploading piece. err: <nil> Dec 27 13:50:58.011194257 (526h21m57.882277302s) [noise=true , address=45.91.116.13:4153 , piece_id=SM6W2CKCVET3QYOHUFJVIXQR5EDHIQ3RKFVVPT4WJLWL2SLHFZWQ, seg_pos={0 0}, upload=461701]: Done uploading piece. err: <nil> Dec 27 13:50:58.037013111 (526h21m57.908096156s) [noise=true , address=45.91.116.5:4298 , piece_id=7CIWYC66SESQGY5RHN2RDABPKSC6YH7RVL2MPXKPBSYUTRK5ZXYQ, seg_pos={0 0}, upload=461701]: Done uploading piece. err: <nil> Dec 27 13:50:58.062523106 (526h21m57.933606141s) [noise=true , address=77.90.31.12:4590 , piece_id=POX7H3SSLNBQNY3VD62NXDYPSDFTBPGWDFZ2FECTVUCGZ3FVTTGA, seg_pos={0 0}, upload=461701]: Done uploading piece. err: <nil> Dec 27 13:50:58.078173908 (526h21m57.949256943s) [noise=true , address=45.91.116.12:4226 , piece_id=HIOB42F74EPG5LXICFS53IMR734NCKC2AOTCYMD6ROITHDMLLRXA, seg_pos={0 0}, upload=461701]: Done uploading piece. err: <nil> Dec 27 13:50:58.078193686 (526h21m57.949276721s) [seg_pos={0 0}, upload=461701]: Segment reached optimal threshold of 13 pieces. Dec 27 13:50:58.078374259 (526h21m57.949457295s) [noise=true , address=77.90.31.11:4554 , piece_id=BLMCQ5DBRGSXBGFCCXOU7YGLALSWSQIKWEVIVD5GNR6H4W52F2BQ, seg_pos={0 0}, upload=461701]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:49524->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:49524->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49524->77.90.31.11:4554: use of closed network connection Dec 27 13:50:58.078380792 (526h21m57.949463837s) [noise=true , address=77.90.31.11:4554 , piece_id=BLMCQ5DBRGSXBGFCCXOU7YGLALSWSQIKWEVIVD5GNR6H4W52F2BQ, seg_pos={0 0}, upload=461701]: Long tail cancellation due to optimal threshold Dec 27 13:50:58.078465042 (526h21m57.949548077s) [noise=true , address=45.91.116.4:4940 , piece_id=6NY4ART5MKDAIOMWIRZP3JZVHJYH56SZGJXOC4TUXQVT3QBV2H7A, seg_pos={0 0}, upload=461701]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 27 13:50:58.078474230 (526h21m57.949557265s) [noise=true , address=45.91.116.4:4940 , piece_id=6NY4ART5MKDAIOMWIRZP3JZVHJYH56SZGJXOC4TUXQVT3QBV2H7A, seg_pos={0 0}, upload=461701]: Long tail cancellation due to optimal threshold Dec 27 13:50:58.078671525 (526h21m57.94975457s) [noise=true , address=145.79.227.4:4187 , piece_id=PI2OUTI5AIJIH4DOENMMBSSKHBXNQZWJHTWOKO7ZHNVIIID4INZQ, seg_pos={0 0}, upload=461701]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: expected piece hash; context canceled; EOF Dec 27 13:50:58.078680783 (526h21m57.949763838s) [noise=true , address=145.79.227.4:4187 , piece_id=PI2OUTI5AIJIH4DOENMMBSSKHBXNQZWJHTWOKO7ZHNVIIID4INZQ, seg_pos={0 0}, upload=461701]: Long tail cancellation due to optimal threshold Dec 27 13:50:58.078703306 (526h21m57.949786351s) [seg_pos={0 0}, upload=461701]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:50:58.078789120 (526h21m57.949872165s) [upload=461701]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:50:58.078795011 (526h21m57.949878046s) [upload=461701]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:50:58.368973500 (526h21m58.240056545s) [upload=461701]: Done uploading segments... Dec 27 13:50:58.368999008 (526h21m58.240082044s) [upload=461701]: Upload finished. err: <nil> Duration: 30.00263984s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..................X...X...X..........X.............XX.......] duration: 52.505261772s trace: https://jaeger.dev.storj.io/trace/3ed1e657d506c4ee FAIL (2025-12-27 13:51:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, Piece ID: HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ) context deadline exceeded; piecestore: (Node ID: 12wXqePK1kK7iBo6wHcbqkRmkED8LGJ3rzQNJAWa14cHYhx1oyB, Piece ID: M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA) context deadline exceeded; piecestore: (Node ID: 12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx, Piece ID: ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ) context deadline exceeded; piecestore: (Node ID: 12UjFkFn2UEvr57LDorcijDCvkBkkKitgZEFx6Yqfk5gMStQvJG, Piece ID: SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ) context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.16:4739 , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.319646322 (526h14m34.190729357s) [noise=true , address=77.90.31.21:4899 , piece_id=SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.319933248 (526h14m34.191016283s) [noise=true , address=45.91.116.12:4234 , piece_id=Y2FIVYLZBQAYBO2TDGIZ4RDQHQJ3WOQH565A7ESIKANQQTQDFQZQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320400588 (526h14m34.191483633s) [noise=true , address=145.79.227.29:4952 , piece_id=VXZLF4OGRIR3MYQMMLKLTD235A55F4YTHB3D4YCVCQUL4FZTSFJQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320407341 (526h14m34.191490386s) [noise=true , address=145.79.227.46:4621 , piece_id=XZ4MKAZAGQKVEJJGSZ5QGT4AT3MFXXNMXICVFCI6SP3E4EVEHZGQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320675030 (526h14m34.191758075s) [noise=true , address=45.91.116.10:4113 , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320802022 (526h14m34.191885067s) [noise=true , address=77.90.31.11:4569 , piece_id=YC2Q3HPJDVY3NEDZNC2Z4KVN6S6DDDGRARASF7KALFAOQ335A7HA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.321169643 (526h14m34.192252678s) [noise=true , address=77.90.31.19:4869 , piece_id=HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.321443483 (526h14m34.192526518s) [noise=true , address=45.91.116.14:4202 , piece_id=M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.321731813 (526h14m34.192814848s) [noise=true , address=77.90.31.18:4819 , piece_id=YS4DXTHLGCPFOKKLBA2DK2BVOAPTQEGVOPBSRCLFV26SVP6YFHGA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.322221776 (526h14m34.193304811s) [noise=true , address=77.90.31.15:4708 , piece_id=3NX6E54XMZXQAQQE27LTWUSEF5GAIF2XOPQFFAP7LOA2PL7KDL3Q, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.322547776 (526h14m34.193630811s) [noise=true , address=145.79.227.40:4079 , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.322908093 (526h14m34.193991138s) [noise=true , address=145.79.227.29:4975 , piece_id=DAMNBVP7DRSW5OAGYRRP4K5SNJ2WEUT7IAXO77KWRHLQEMBNERZQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.323267968 (526h14m34.194351003s) [noise=true , address=145.79.227.30:4787 , piece_id=OB2OTY3ZEHIUFYM6FY7HIS7HQ353Y2HL6XU4476HXOD26QVA4DPQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.323618165 (526h14m34.19470121s) [noise=true , address=64.49.13.169:4710 , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.324084893 (526h14m34.195167928s) [noise=true , address=193.149.164.143:4694 , piece_id=ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:35.783540136 (526h14m35.654623181s) [upload=461595]: Next returned nil segment Dec 27 13:43:35.783555856 (526h14m35.654638891s) [upload=461595]: Waiting for error group managing segments... Dec 27 13:43:36.192287250 (526h14m36.063370296s) [noise=true , address=145.79.227.29:4952 , piece_id=VXZLF4OGRIR3MYQMMLKLTD235A55F4YTHB3D4YCVCQUL4FZTSFJQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.200200289 (526h14m36.071283334s) [noise=true , address=145.79.227.46:4621 , piece_id=XZ4MKAZAGQKVEJJGSZ5QGT4AT3MFXXNMXICVFCI6SP3E4EVEHZGQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.238087532 (526h14m36.109170577s) [noise=true , address=145.79.227.29:4975 , piece_id=DAMNBVP7DRSW5OAGYRRP4K5SNJ2WEUT7IAXO77KWRHLQEMBNERZQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.379631346 (526h14m36.250714391s) [noise=true , address=77.90.31.19:4869 , piece_id=HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.382945286 (526h14m36.254028321s) [noise=true , address=145.79.227.34:4909 , piece_id=Z6RF5PSOGOZMOXPQYXHMXGFXIELVXXVU4NXXRTAQCIELQXLMVXBQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.400290103 (526h14m36.271373138s) [noise=true , address=77.90.31.18:4819 , piece_id=YS4DXTHLGCPFOKKLBA2DK2BVOAPTQEGVOPBSRCLFV26SVP6YFHGA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.436707208 (526h14m36.307790243s) [noise=true , address=45.91.116.14:4202 , piece_id=M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.487651475 (526h14m36.35873451s) [noise=true , address=45.91.116.12:4234 , piece_id=Y2FIVYLZBQAYBO2TDGIZ4RDQHQJ3WOQH565A7ESIKANQQTQDFQZQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.500850769 (526h14m36.371933814s) [noise=true , address=77.90.31.11:4569 , piece_id=YC2Q3HPJDVY3NEDZNC2Z4KVN6S6DDDGRARASF7KALFAOQ335A7HA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.518828551 (526h14m36.389911586s) [noise=true , address=77.90.31.15:4708 , piece_id=3NX6E54XMZXQAQQE27LTWUSEF5GAIF2XOPQFFAP7LOA2PL7KDL3Q, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.051607243 (526h14m36.922690289s) [noise=true , address=145.79.227.44:4228 , piece_id=6SAYEAOTTCOUFPU75C7YRIULWJGXL4PK2YKCKFB47EUBWMRQQQOQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.096826909 (526h14m36.967909954s) [noise=true , address=145.79.227.30:4787 , piece_id=OB2OTY3ZEHIUFYM6FY7HIS7HQ353Y2HL6XU4476HXOD26QVA4DPQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.140738995 (526h14m37.01182203s) [noise=true , address=77.90.31.21:4899 , piece_id=SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.323613990 (526h14m37.194697035s) [noise=true , address=193.149.164.143:4694 , piece_id=ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.323649618 (526h14m37.194732663s) [seg_pos={0 0}, upload=461595]: Segment reached optimal threshold of 14 pieces. Dec 27 13:43:37.323852865 (526h14m37.1949359s) [noise=true , address=45.91.116.10:4113 , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12puRMny4nbgv2by4tAbXcBBSgG2d6BfwcDDRDcFzEdjSVUJxtd): protocol: expected piece hash; context canceled; EOF Dec 27 13:43:37.323860349 (526h14m37.194943384s) [noise=true , address=45.91.116.10:4113 , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324005135 (526h14m37.1950882s) [noise=true , address=145.79.227.40:4079 , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RYKzn5zVSmxVbWwnBdX8oqtG1WE2MhPeEiWLf5dZeFAut6s2Y): protocol: expected piece hash; context canceled; EOF Dec 27 13:43:37.324022659 (526h14m37.195105704s) [noise=true , address=145.79.227.40:4079 , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324143378 (526h14m37.195226413s) [noise=true , address=77.90.31.16:4739 , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129B3N7yjJvwCzoVLTqWsPbcLQ5TQUyAzg1RCkbSAVKPWYTG8eb): protocol: expected piece hash; context canceled; EOF Dec 27 13:43:37.324154520 (526h14m37.195237555s) [noise=true , address=77.90.31.16:4739 , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324213442 (526h14m37.195296487s) [noise=true , address=64.49.13.169:4710 , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection Dec 27 13:43:37.324249360 (526h14m37.195332405s) [noise=true , address=64.49.13.169:4710 , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324275360 (526h14m37.195358405s) [seg_pos={0 0}, upload=461595]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 13:43:37.324374018 (526h14m37.195457053s) [upload=461595]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:43:37.324381291 (526h14m37.195464337s) [upload=461595]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:43:37.679036406 (526h14m37.550119451s) [upload=461595]: Done uploading segments... Dec 27 13:43:37.679056985 (526h14m37.55014003s) [upload=461595]: Upload finished. err: <nil> Duration: 52.505331463s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [XX......X..X...X.XXX....X...X...X...XX.......X....X..XX.X.XX] duration: 30.004669591s trace: https://jaeger.dev.storj.io/trace/12958016913e3ef7 FAIL (2025-12-27 13:50:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 27 13:50:25.289481291 (526h21m25.160564326s) [upload=461692]: Starting upload Dec 27 13:50:25.289572655 (526h21m25.16065572s) [upload=461692]: Done starting upload Dec 27 13:50:25.289697152 (526h21m25.160780187s) [upload=461692]: Uploading segments... Dec 27 13:50:25.289908865 (526h21m25.160991911s) [upload=461692]: Got next segment. Inline: false Dec 27 13:50:25.290005390 (526h21m25.161088445s) [upload=461692]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:50:25.290014927 (526h21m25.161097982s) [upload=461692]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:50:25.537495353 (526h21m25.408578398s) [seg_pos={0 0}, upload=461692]: Begin upload segment... Dec 27 13:50:25.537554414 (526h21m25.408637459s) [seg_pos={0 0}, upload=461692]: Done begin upload segment. Dec 27 13:50:25.537580073 (526h21m25.408663118s) [noise=true , address=145.79.227.36:4518 , piece_id=BWEKPAQFYN5O3PO3WKVCLBEZ4Y6O3I57NOB4ZQQTA3437QCIHJMQ, seg_pos={0 0}, upload=461692]: Uploading piece... Dec 27 13:50:25.537591294 (526h21m25.40867434s) [noise=true , address=145.79.227.32:4834 , piece_id=436YK7TTIEVFMWSHKDBWI5IQDXDH77HNAOQX64HZKMFVMSB2YGHQ, seg_pos={0 0}, upload=461692]: Uploading piece... Dec 27 13:50:25.538536776 (526h21m25.409619821s) [noise=true , address=145.79.227.48:4303 , piece_id=XUSDOOP3IWHLJKJO774BYLJ4NWQDQ35E5UN2RJTCFXQNQ5NGZZSQ, seg_pos={0 0}, upload=461692]: Uploading piece... Dec 27 13:50:25.538668536 (526h21m25.409751581s) [noise=true , address=145.79.227.41:4085 , piece_id=JT37YFH2YNJIOJIUMUILDVOFZPWNW4DXNYM3VNBJSKM3P254ONPQ, seg_pos={0 0}, upload=461692]: Uploading piece... Dec 27 13:50:25.539200529 (526h21m25.410283575s) [noise=true , address=145.79.227.45:4231 , piece_id=NB7XATBNXHQVCMCPHQBRE2B3M23L7L4PIHXDCHHCMDFLUJNKHT5Q, seg_pos={0 0}, upload=461692]: Uploading piece... Dec 27 13:50:25.539302223 (526h21m25.410385268s) [noise=true , address=45.91.116.12:4226 , piece_id=S6TBABHQ7DVEWTXPM2EXXA75P77BEYL6PKYRXL2BIGXNHGGEJWNA, seg_pos={0 0}, upload=461692]: Uploading piece... Dec 27 13:50:25.539716732 (526h21m25.410799787s) [noise=true , address=145.79.227.5:4221 , piece_id=MIXIK7QA6JNGYF6DV74QMRLMGFYVK2XQNYD3CAIAE5Y37F5OHMOA, seg_pos={0 0}, upload=461692]: Uploading piece... Dec 27 13:50:25.539945868 (526h21m25.411028913s) [noise=true , address=45.91.116.10:4083 , piece_id=5SHRX6BZ2OQYQPRMI24GPPTYIEYGVQAC5RAIVBCAQDYMCCQ3OQDQ, seg_pos={0 0}, upload=461692]: Uploading piece... Dec 27 13:50:25.540140439 (526h21m25.411223484s) [noise=true , address=145.79.227.42:4123 , piece_id=LSY2GTLXAMVOY3PIYW7M75CQIXP7IXGNUKVAC2FEIR4NMMWQZPMQ, seg_pos={0 0}, upload=461692]: Uploading piece... Dec 27 13:50:25.540538717 (526h21m25.411621752s) [noise=true , address=45.91.116.4:4939 , piece_id=TUOP2QK73VHGNNGEOST2TQBO57IWJDNFHQ63SCDDUOTC6T2FMR5Q, seg_pos={0 0}, upload=461692]: Uploading piece... Dec 27 13:50:25.540732086 (526h21m25.411815131s) [noise=true , address=45.91.116.5:4297 , piece_id=ZRQ3ZWEUJ3PYH4YEZXT2AV7FBJTUMJGWZGR3TZ7ZIE2RV2ARVTFA, seg_pos={0 0}, upload=461692]: Uploading piece... Dec 27 13:50:25.541209745 (526h21m25.41229279s) [noise=true , address=145.79.227.7:4294 , piece_id=DUUOPLMREGN3QK7MQYHOJUVQZA7EBFOWMN462VLQCWGEABIC62IA, seg_pos={0 0}, upload=461692]: Uploading piece... Dec 27 13:50:25.541647378 (526h21m25.412730424s) [noise=true , address=45.91.116.7:4011 , piece_id=TE4ZAK3IEJ6LKNMIOXWZL6LVR55JH4XOEBVV46PNGD6MCATRBTQA, seg_pos={0 0}, upload=461692]: Uploading piece... Dec 27 13:50:25.542281215 (526h21m25.41336426s) [noise=true , address=145.79.227.11:4732 , piece_id=4D5KM5WSZQ6RDWBNUTNIU7PIXJSB2QXKYVW2IJFFDVGY5N3N64IA, seg_pos={0 0}, upload=461692]: Uploading piece... Dec 27 13:50:25.542811054 (526h21m25.413894099s) [noise=true , address=77.90.31.13:4629 , piece_id=ZTWSKBSREXTK44SA7JJZT7QUIXHUQOF6X3MAOTE4CGHGHQMY642Q, seg_pos={0 0}, upload=461692]: Uploading piece... Dec 27 13:50:25.543474086 (526h21m25.414557131s) [noise=true , address=145.79.227.16:4911 , piece_id=VXOUXFF7JHHBYPISOGFEJXHEPCPGD6KSPWOTWRCMOLRR7KZJ3ILA, seg_pos={0 0}, upload=461692]: Uploading piece... Dec 27 13:50:27.247617097 (526h21m27.118700142s) [upload=461692]: Next returned nil segment Dec 27 13:50:27.247630953 (526h21m27.118713988s) [upload=461692]: Waiting for error group managing segments... Dec 27 13:50:27.533508413 (526h21m27.404591458s) [noise=true , address=145.79.227.7:4294 , piece_id=DUUOPLMREGN3QK7MQYHOJUVQZA7EBFOWMN462VLQCWGEABIC62IA, seg_pos={0 0}, upload=461692]: Done uploading piece. err: <nil> Dec 27 13:50:27.534322443 (526h21m27.405405488s) [noise=true , address=145.79.227.16:4911 , piece_id=VXOUXFF7JHHBYPISOGFEJXHEPCPGD6KSPWOTWRCMOLRR7KZJ3ILA, seg_pos={0 0}, upload=461692]: Done uploading piece. err: <nil> Dec 27 13:50:27.537305573 (526h21m27.408388618s) [noise=true , address=145.79.227.42:4123 , piece_id=LSY2GTLXAMVOY3PIYW7M75CQIXP7IXGNUKVAC2FEIR4NMMWQZPMQ, seg_pos={0 0}, upload=461692]: Done uploading piece. err: <nil> Dec 27 13:50:27.543595220 (526h21m27.414678265s) [noise=true , address=145.79.227.5:4221 , piece_id=MIXIK7QA6JNGYF6DV74QMRLMGFYVK2XQNYD3CAIAE5Y37F5OHMOA, seg_pos={0 0}, upload=461692]: Done uploading piece. err: <nil> Dec 27 13:50:27.548319225 (526h21m27.41940227s) [noise=true , address=145.79.227.36:4518 , piece_id=BWEKPAQFYN5O3PO3WKVCLBEZ4Y6O3I57NOB4ZQQTA3437QCIHJMQ, seg_pos={0 0}, upload=461692]: Done uploading piece. err: <nil> Dec 27 13:50:27.548619647 (526h21m27.419702682s) [noise=true , address=145.79.227.45:4231 , piece_id=NB7XATBNXHQVCMCPHQBRE2B3M23L7L4PIHXDCHHCMDFLUJNKHT5Q, seg_pos={0 0}, upload=461692]: Done uploading piece. err: <nil> Dec 27 13:50:27.551571819 (526h21m27.422654854s) [noise=true , address=145.79.227.41:4085 , piece_id=JT37YFH2YNJIOJIUMUILDVOFZPWNW4DXNYM3VNBJSKM3P254ONPQ, seg_pos={0 0}, upload=461692]: Done uploading piece. err: <nil> Dec 27 13:50:27.552121175 (526h21m27.42320422s) [noise=true , address=145.79.227.11:4732 , piece_id=4D5KM5WSZQ6RDWBNUTNIU7PIXJSB2QXKYVW2IJFFDVGY5N3N64IA, seg_pos={0 0}, upload=461692]: Done uploading piece. err: <nil> Dec 27 13:50:27.553625169 (526h21m27.424708204s) [noise=true , address=145.79.227.32:4834 , piece_id=436YK7TTIEVFMWSHKDBWI5IQDXDH77HNAOQX64HZKMFVMSB2YGHQ, seg_pos={0 0}, upload=461692]: Done uploading piece. err: <nil> Dec 27 13:50:27.574617883 (526h21m27.445700928s) [noise=true , address=145.79.227.48:4303 , piece_id=XUSDOOP3IWHLJKJO774BYLJ4NWQDQ35E5UN2RJTCFXQNQ5NGZZSQ, seg_pos={0 0}, upload=461692]: Done uploading piece. err: <nil> Dec 27 13:50:27.831061044 (526h21m27.702144079s) [noise=true , address=45.91.116.5:4297 , piece_id=ZRQ3ZWEUJ3PYH4YEZXT2AV7FBJTUMJGWZGR3TZ7ZIE2RV2ARVTFA, seg_pos={0 0}, upload=461692]: Done uploading piece. err: <nil> Dec 27 13:50:27.861136573 (526h21m27.732219618s) [noise=true , address=45.91.116.10:4083 , piece_id=5SHRX6BZ2OQYQPRMI24GPPTYIEYGVQAC5RAIVBCAQDYMCCQ3OQDQ, seg_pos={0 0}, upload=461692]: Done uploading piece. err: <nil> Dec 27 13:50:28.553773732 (526h21m28.424856777s) [noise=true , address=45.91.116.7:4011 , piece_id=TE4ZAK3IEJ6LKNMIOXWZL6LVR55JH4XOEBVV46PNGD6MCATRBTQA, seg_pos={0 0}, upload=461692]: Done uploading piece. err: <nil> Dec 27 13:50:28.554117036 (526h21m28.425200101s) [seg_pos={0 0}, upload=461692]: Segment reached optimal threshold of 13 pieces. Dec 27 13:50:28.554346654 (526h21m28.425429699s) [noise=true , address=77.90.31.13:4629 , piece_id=ZTWSKBSREXTK44SA7JJZT7QUIXHUQOF6X3MAOTE4CGHGHQMY642Q, seg_pos={0 0}, upload=461692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 27 13:50:28.554355961 (526h21m28.425438996s) [noise=true , address=77.90.31.13:4629 , piece_id=ZTWSKBSREXTK44SA7JJZT7QUIXHUQOF6X3MAOTE4CGHGHQMY642Q, seg_pos={0 0}, upload=461692]: Long tail cancellation due to optimal threshold Dec 27 13:50:28.554500226 (526h21m28.425583271s) [noise=true , address=45.91.116.12:4226 , piece_id=S6TBABHQ7DVEWTXPM2EXXA75P77BEYL6PKYRXL2BIGXNHGGEJWNA, seg_pos={0 0}, upload=461692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:49760->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:49760->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49760->45.91.116.12:4226: use of closed network connection Dec 27 13:50:28.554525614 (526h21m28.42560865s) [noise=true , address=45.91.116.12:4226 , piece_id=S6TBABHQ7DVEWTXPM2EXXA75P77BEYL6PKYRXL2BIGXNHGGEJWNA, seg_pos={0 0}, upload=461692]: Long tail cancellation due to optimal threshold Dec 27 13:50:28.554572103 (526h21m28.425655138s) [noise=true , address=45.91.116.4:4939 , piece_id=TUOP2QK73VHGNNGEOST2TQBO57IWJDNFHQ63SCDDUOTC6T2FMR5Q, seg_pos={0 0}, upload=461692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 27 13:50:28.554578616 (526h21m28.425661651s) [noise=true , address=45.91.116.4:4939 , piece_id=TUOP2QK73VHGNNGEOST2TQBO57IWJDNFHQ63SCDDUOTC6T2FMR5Q, seg_pos={0 0}, upload=461692]: Long tail cancellation due to optimal threshold Dec 27 13:50:28.554746855 (526h21m28.4258299s) [seg_pos={0 0}, upload=461692]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:50:28.554840604 (526h21m28.425923639s) [upload=461692]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:50:28.554846345 (526h21m28.42592938s) [upload=461692]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:50:28.849188513 (526h21m28.720271548s) [upload=461692]: Done uploading segments... Dec 27 13:50:28.849206668 (526h21m28.720289703s) [upload=461692]: Upload finished. err: <nil> Duration: 30.0047373s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.XX......X..X...X.XXX....X...X...X...XX.......X....X..XX.X.X] duration: 30.002508924s trace: https://jaeger.dev.storj.io/trace/7aad7c46c5b40182 FAIL (2025-12-27 13:50:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 13:49:55.036654820 (526h20m54.907737855s) [upload=461684]: Starting upload Dec 27 13:49:55.036787763 (526h20m54.907870809s) [upload=461684]: Done starting upload Dec 27 13:49:55.036902812 (526h20m54.907985857s) [upload=461684]: Uploading segments... Dec 27 13:49:55.037066123 (526h20m54.908149169s) [upload=461684]: Got next segment. Inline: false Dec 27 13:49:55.037115166 (526h20m54.908198202s) [upload=461684]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:49:55.037122360 (526h20m54.908205405s) [upload=461684]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:49:55.283393253 (526h20m55.154476298s) [seg_pos={0 0}, upload=461684]: Begin upload segment... Dec 27 13:49:55.283445582 (526h20m55.154528617s) [seg_pos={0 0}, upload=461684]: Done begin upload segment. Dec 27 13:49:55.283475539 (526h20m55.154558574s) [noise=true , address=45.91.116.9:4262 , piece_id=QBINWWOQTBSXPG2Z2MRZ5L73IOR57HKT7JQ54A7SSIX5N6AC3EEA, seg_pos={0 0}, upload=461684]: Uploading piece... Dec 27 13:49:55.283506498 (526h20m55.154589533s) [noise=true , address=45.91.116.14:4190 , piece_id=BNWAOFKGGRR66EK26KBDBTEEADXNWT5D6QCY5TJ4KRQRQUDM2WJQ, seg_pos={0 0}, upload=461684]: Uploading piece... Dec 27 13:49:55.283965953 (526h20m55.155048988s) [noise=true , address=145.79.227.43:4158 , piece_id=F2IJZHRCHI6JELF2CT7V5DREZHASJKYS342FQQRPD6UZGU6YYTOA, seg_pos={0 0}, upload=461684]: Uploading piece... Dec 27 13:49:55.284160383 (526h20m55.155243418s) [noise=true , address=45.91.116.12:4225 , piece_id=TCRLMCM42UGGAP7ZGTF6ABLQBLEHOCHJAGVEJ3HPDDSLBJEYIL7Q, seg_pos={0 0}, upload=461684]: Uploading piece... Dec 27 13:49:55.284427501 (526h20m55.155510546s) [noise=true , address=145.79.227.46:4603 , piece_id=J2NCQ3M576A74SS5AYBNQJYLNJNEHHSQPPPYFZG3EPG4IDPSC5GQ, seg_pos={0 0}, upload=461684]: Uploading piece... Dec 27 13:49:55.284683029 (526h20m55.155766074s) [noise=true , address=145.79.227.36:4518 , piece_id=MV4LQRCCV6PKGSZPNETNRGU74HT2PXVKCHGO5TWMALYQWDNRAWFQ, seg_pos={0 0}, upload=461684]: Uploading piece... Dec 27 13:49:55.284949195 (526h20m55.15603224s) [noise=true , address=77.90.31.11:4553 , piece_id=4WNINFEJHZIZQW6CICXKKNG3B5UK3AI3OGHEYOPQI3BM2XD2VFIQ, seg_pos={0 0}, upload=461684]: Uploading piece... Dec 27 13:49:55.284984542 (526h20m55.156067587s) [noise=true , address=145.79.227.38:4979 , piece_id=2DQPHYZ76QROSXC7LWMIOSL2KKVIY23ZHB7AXKWTN2CQZAKCSH5Q, seg_pos={0 0}, upload=461684]: Uploading piece... Dec 27 13:49:55.285265748 (526h20m55.156348784s) [noise=true , address=145.79.227.4:4187 , piece_id=MTZSX7JYOU36LOPR7T5SSQM6LQQ3BXRAYJ64QC7E6JM7G42AXYNQ, seg_pos={0 0}, upload=461684]: Uploading piece... Dec 27 13:49:55.285464115 (526h20m55.156547161s) [noise=true , address=45.91.116.13:4153 , piece_id=NMTFJICSAEA6CUQ3AF637CQK55CYOSR6RU5SNCNBVZW3AQIQTLQQ, seg_pos={0 0}, upload=461684]: Uploading piece... Dec 27 13:49:55.285522337 (526h20m55.156605372s) [noise=true , address=145.79.227.17:4971 , piece_id=KNACILXFURUKBCCKOTH73WAQPWV5QXBZMZ7KYID3RLKI3STPYUSA, seg_pos={0 0}, upload=461684]: Uploading piece... Dec 27 13:49:55.285838669 (526h20m55.156921704s) [noise=true , address=45.91.116.11:4117 , piece_id=DEJDVTH4PX4SEBSC7D43WME32TGKWKRRTRWCL66FK5OUNCN2IGOA, seg_pos={0 0}, upload=461684]: Uploading piece... Dec 27 13:49:55.286261574 (526h20m55.157344609s) [noise=true , address=145.79.227.37:4566 , piece_id=O7M6MLYQYR3LP3RQGUFAZC4FMH3ILSNHQULKLMXSDHNCQ6J7B5EQ, seg_pos={0 0}, upload=461684]: Uploading piece... Dec 27 13:49:55.286782366 (526h20m55.157865411s) [noise=true , address=145.79.227.11:4729 , piece_id=QA2ISLNYWZUXQG3GMSCSJBMU2EQJFLKE3M2N75RQSYJZPIIIRBMA, seg_pos={0 0}, upload=461684]: Uploading piece... Dec 27 13:49:55.287783082 (526h20m55.158866127s) [noise=true , address=145.79.227.42:4122 , piece_id=L5XYFQ4NSDXABAYCQCQRZ64LUCMYWUDYYYPVORDGRYLFK4PGCWFQ, seg_pos={0 0}, upload=461684]: Uploading piece... Dec 27 13:49:55.288164498 (526h20m55.159247533s) [noise=true , address=145.79.227.47:4266 , piece_id=L6ALCLBYK5BBEWA6MRAWCKLAZZXIPFUOOHGYBXKJLFY7KIBWUNAQ, seg_pos={0 0}, upload=461684]: Uploading piece... Dec 27 13:49:57.000742199 (526h20m56.871825234s) [upload=461684]: Next returned nil segment Dec 27 13:49:57.000755744 (526h20m56.871838779s) [upload=461684]: Waiting for error group managing segments... Dec 27 13:49:57.278632429 (526h20m57.149715474s) [noise=true , address=145.79.227.17:4971 , piece_id=KNACILXFURUKBCCKOTH73WAQPWV5QXBZMZ7KYID3RLKI3STPYUSA, seg_pos={0 0}, upload=461684]: Done uploading piece. err: <nil> Dec 27 13:49:57.285720867 (526h20m57.156803912s) [noise=true , address=145.79.227.36:4518 , piece_id=MV4LQRCCV6PKGSZPNETNRGU74HT2PXVKCHGO5TWMALYQWDNRAWFQ, seg_pos={0 0}, upload=461684]: Done uploading piece. err: <nil> Dec 27 13:49:57.287409772 (526h20m57.158492807s) [noise=true , address=145.79.227.37:4566 , piece_id=O7M6MLYQYR3LP3RQGUFAZC4FMH3ILSNHQULKLMXSDHNCQ6J7B5EQ, seg_pos={0 0}, upload=461684]: Done uploading piece. err: <nil> Dec 27 13:49:57.292882233 (526h20m57.163965299s) [noise=true , address=145.79.227.38:4979 , piece_id=2DQPHYZ76QROSXC7LWMIOSL2KKVIY23ZHB7AXKWTN2CQZAKCSH5Q, seg_pos={0 0}, upload=461684]: Done uploading piece. err: <nil> Dec 27 13:49:57.293552259 (526h20m57.164635314s) [noise=true , address=145.79.227.46:4603 , piece_id=J2NCQ3M576A74SS5AYBNQJYLNJNEHHSQPPPYFZG3EPG4IDPSC5GQ, seg_pos={0 0}, upload=461684]: Done uploading piece. err: <nil> Dec 27 13:49:57.299064365 (526h20m57.1701474s) [noise=true , address=145.79.227.11:4729 , piece_id=QA2ISLNYWZUXQG3GMSCSJBMU2EQJFLKE3M2N75RQSYJZPIIIRBMA, seg_pos={0 0}, upload=461684]: Done uploading piece. err: <nil> Dec 27 13:49:57.314668188 (526h20m57.185751233s) [noise=true , address=145.79.227.42:4122 , piece_id=L5XYFQ4NSDXABAYCQCQRZ64LUCMYWUDYYYPVORDGRYLFK4PGCWFQ, seg_pos={0 0}, upload=461684]: Done uploading piece. err: <nil> Dec 27 13:49:57.323423409 (526h20m57.194506444s) [noise=true , address=145.79.227.47:4266 , piece_id=L6ALCLBYK5BBEWA6MRAWCKLAZZXIPFUOOHGYBXKJLFY7KIBWUNAQ, seg_pos={0 0}, upload=461684]: Done uploading piece. err: <nil> Dec 27 13:49:57.512028846 (526h20m57.383111881s) [noise=true , address=145.79.227.4:4187 , piece_id=MTZSX7JYOU36LOPR7T5SSQM6LQQ3BXRAYJ64QC7E6JM7G42AXYNQ, seg_pos={0 0}, upload=461684]: Done uploading piece. err: <nil> Dec 27 13:49:57.624076875 (526h20m57.49515992s) [noise=true , address=77.90.31.11:4553 , piece_id=4WNINFEJHZIZQW6CICXKKNG3B5UK3AI3OGHEYOPQI3BM2XD2VFIQ, seg_pos={0 0}, upload=461684]: Done uploading piece. err: <nil> Dec 27 13:49:57.638779070 (526h20m57.509862105s) [noise=true , address=45.91.116.12:4225 , piece_id=TCRLMCM42UGGAP7ZGTF6ABLQBLEHOCHJAGVEJ3HPDDSLBJEYIL7Q, seg_pos={0 0}, upload=461684]: Done uploading piece. err: <nil> Dec 27 13:49:57.680542054 (526h20m57.551625089s) [noise=true , address=45.91.116.11:4117 , piece_id=DEJDVTH4PX4SEBSC7D43WME32TGKWKRRTRWCL66FK5OUNCN2IGOA, seg_pos={0 0}, upload=461684]: Done uploading piece. err: <nil> Dec 27 13:49:58.269835590 (526h20m58.140918625s) [noise=true , address=45.91.116.14:4190 , piece_id=BNWAOFKGGRR66EK26KBDBTEEADXNWT5D6QCY5TJ4KRQRQUDM2WJQ, seg_pos={0 0}, upload=461684]: Done uploading piece. err: <nil> Dec 27 13:49:58.269852892 (526h20m58.140935927s) [seg_pos={0 0}, upload=461684]: Segment reached optimal threshold of 13 pieces. Dec 27 13:49:58.270086076 (526h20m58.141169121s) [noise=true , address=145.79.227.43:4158 , piece_id=F2IJZHRCHI6JELF2CT7V5DREZHASJKYS342FQQRPD6UZGU6YYTOA, seg_pos={0 0}, upload=461684]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 27 13:49:58.270092659 (526h20m58.141175694s) [noise=true , address=145.79.227.43:4158 , piece_id=F2IJZHRCHI6JELF2CT7V5DREZHASJKYS342FQQRPD6UZGU6YYTOA, seg_pos={0 0}, upload=461684]: Long tail cancellation due to optimal threshold Dec 27 13:49:58.270258795 (526h20m58.14134185s) [noise=true , address=45.91.116.9:4262 , piece_id=QBINWWOQTBSXPG2Z2MRZ5L73IOR57HKT7JQ54A7SSIX5N6AC3EEA, seg_pos={0 0}, upload=461684]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 27 13:49:58.270275027 (526h20m58.141358082s) [noise=true , address=45.91.116.9:4262 , piece_id=QBINWWOQTBSXPG2Z2MRZ5L73IOR57HKT7JQ54A7SSIX5N6AC3EEA, seg_pos={0 0}, upload=461684]: Long tail cancellation due to optimal threshold Dec 27 13:49:58.270335030 (526h20m58.141418065s) [noise=true , address=45.91.116.13:4153 , piece_id=NMTFJICSAEA6CUQ3AF637CQK55CYOSR6RU5SNCNBVZW3AQIQTLQQ, seg_pos={0 0}, upload=461684]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 27 13:49:58.270341623 (526h20m58.141424658s) [noise=true , address=45.91.116.13:4153 , piece_id=NMTFJICSAEA6CUQ3AF637CQK55CYOSR6RU5SNCNBVZW3AQIQTLQQ, seg_pos={0 0}, upload=461684]: Long tail cancellation due to optimal threshold Dec 27 13:49:58.270358024 (526h20m58.141441059s) [seg_pos={0 0}, upload=461684]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:49:58.270479014 (526h20m58.141562049s) [upload=461684]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:49:58.270484756 (526h20m58.141567791s) [upload=461684]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:49:58.570254920 (526h20m58.441337965s) [upload=461684]: Done uploading segments... Dec 27 13:49:58.570273956 (526h20m58.441356991s) [upload=461684]: Upload finished. err: <nil> Duration: 30.002533881s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [XX.X.......XXXXX..XXXXX....XX....X.X.X.X.X..XXX.XX.XX.X.XX.X] duration: 5.001013857s trace: https://jaeger.dev.storj.io/trace/450bc773be1f59c4 FAIL (2025-12-27 13:50:22 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001032292s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...................X...X...X..........X.............XX......] duration: 47.776882069s trace: https://jaeger.dev.storj.io/trace/3ed1e657d506c4ee FAIL (2025-12-27 13:49:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, Piece ID: HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ) context deadline exceeded; piecestore: (Node ID: 12wXqePK1kK7iBo6wHcbqkRmkED8LGJ3rzQNJAWa14cHYhx1oyB, Piece ID: M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA) context deadline exceeded; piecestore: (Node ID: 12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx, Piece ID: ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ) context deadline exceeded; piecestore: (Node ID: 12UjFkFn2UEvr57LDorcijDCvkBkkKitgZEFx6Yqfk5gMStQvJG, Piece ID: SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ) context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.16:4739 , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.319646322 (526h14m34.190729357s) [noise=true , address=77.90.31.21:4899 , piece_id=SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.319933248 (526h14m34.191016283s) [noise=true , address=45.91.116.12:4234 , piece_id=Y2FIVYLZBQAYBO2TDGIZ4RDQHQJ3WOQH565A7ESIKANQQTQDFQZQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320400588 (526h14m34.191483633s) [noise=true , address=145.79.227.29:4952 , piece_id=VXZLF4OGRIR3MYQMMLKLTD235A55F4YTHB3D4YCVCQUL4FZTSFJQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320407341 (526h14m34.191490386s) [noise=true , address=145.79.227.46:4621 , piece_id=XZ4MKAZAGQKVEJJGSZ5QGT4AT3MFXXNMXICVFCI6SP3E4EVEHZGQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320675030 (526h14m34.191758075s) [noise=true , address=45.91.116.10:4113 , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.320802022 (526h14m34.191885067s) [noise=true , address=77.90.31.11:4569 , piece_id=YC2Q3HPJDVY3NEDZNC2Z4KVN6S6DDDGRARASF7KALFAOQ335A7HA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.321169643 (526h14m34.192252678s) [noise=true , address=77.90.31.19:4869 , piece_id=HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.321443483 (526h14m34.192526518s) [noise=true , address=45.91.116.14:4202 , piece_id=M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.321731813 (526h14m34.192814848s) [noise=true , address=77.90.31.18:4819 , piece_id=YS4DXTHLGCPFOKKLBA2DK2BVOAPTQEGVOPBSRCLFV26SVP6YFHGA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.322221776 (526h14m34.193304811s) [noise=true , address=77.90.31.15:4708 , piece_id=3NX6E54XMZXQAQQE27LTWUSEF5GAIF2XOPQFFAP7LOA2PL7KDL3Q, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.322547776 (526h14m34.193630811s) [noise=true , address=145.79.227.40:4079 , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.322908093 (526h14m34.193991138s) [noise=true , address=145.79.227.29:4975 , piece_id=DAMNBVP7DRSW5OAGYRRP4K5SNJ2WEUT7IAXO77KWRHLQEMBNERZQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.323267968 (526h14m34.194351003s) [noise=true , address=145.79.227.30:4787 , piece_id=OB2OTY3ZEHIUFYM6FY7HIS7HQ353Y2HL6XU4476HXOD26QVA4DPQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.323618165 (526h14m34.19470121s) [noise=true , address=64.49.13.169:4710 , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:34.324084893 (526h14m34.195167928s) [noise=true , address=193.149.164.143:4694 , piece_id=ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ, seg_pos={0 0}, upload=461595]: Uploading piece... Dec 27 13:43:35.783540136 (526h14m35.654623181s) [upload=461595]: Next returned nil segment Dec 27 13:43:35.783555856 (526h14m35.654638891s) [upload=461595]: Waiting for error group managing segments... Dec 27 13:43:36.192287250 (526h14m36.063370296s) [noise=true , address=145.79.227.29:4952 , piece_id=VXZLF4OGRIR3MYQMMLKLTD235A55F4YTHB3D4YCVCQUL4FZTSFJQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.200200289 (526h14m36.071283334s) [noise=true , address=145.79.227.46:4621 , piece_id=XZ4MKAZAGQKVEJJGSZ5QGT4AT3MFXXNMXICVFCI6SP3E4EVEHZGQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.238087532 (526h14m36.109170577s) [noise=true , address=145.79.227.29:4975 , piece_id=DAMNBVP7DRSW5OAGYRRP4K5SNJ2WEUT7IAXO77KWRHLQEMBNERZQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.379631346 (526h14m36.250714391s) [noise=true , address=77.90.31.19:4869 , piece_id=HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.382945286 (526h14m36.254028321s) [noise=true , address=145.79.227.34:4909 , piece_id=Z6RF5PSOGOZMOXPQYXHMXGFXIELVXXVU4NXXRTAQCIELQXLMVXBQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.400290103 (526h14m36.271373138s) [noise=true , address=77.90.31.18:4819 , piece_id=YS4DXTHLGCPFOKKLBA2DK2BVOAPTQEGVOPBSRCLFV26SVP6YFHGA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.436707208 (526h14m36.307790243s) [noise=true , address=45.91.116.14:4202 , piece_id=M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.487651475 (526h14m36.35873451s) [noise=true , address=45.91.116.12:4234 , piece_id=Y2FIVYLZBQAYBO2TDGIZ4RDQHQJ3WOQH565A7ESIKANQQTQDFQZQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.500850769 (526h14m36.371933814s) [noise=true , address=77.90.31.11:4569 , piece_id=YC2Q3HPJDVY3NEDZNC2Z4KVN6S6DDDGRARASF7KALFAOQ335A7HA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:36.518828551 (526h14m36.389911586s) [noise=true , address=77.90.31.15:4708 , piece_id=3NX6E54XMZXQAQQE27LTWUSEF5GAIF2XOPQFFAP7LOA2PL7KDL3Q, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.051607243 (526h14m36.922690289s) [noise=true , address=145.79.227.44:4228 , piece_id=6SAYEAOTTCOUFPU75C7YRIULWJGXL4PK2YKCKFB47EUBWMRQQQOQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.096826909 (526h14m36.967909954s) [noise=true , address=145.79.227.30:4787 , piece_id=OB2OTY3ZEHIUFYM6FY7HIS7HQ353Y2HL6XU4476HXOD26QVA4DPQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.140738995 (526h14m37.01182203s) [noise=true , address=77.90.31.21:4899 , piece_id=SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.323613990 (526h14m37.194697035s) [noise=true , address=193.149.164.143:4694 , piece_id=ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil> Dec 27 13:43:37.323649618 (526h14m37.194732663s) [seg_pos={0 0}, upload=461595]: Segment reached optimal threshold of 14 pieces. Dec 27 13:43:37.323852865 (526h14m37.1949359s) [noise=true , address=45.91.116.10:4113 , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12puRMny4nbgv2by4tAbXcBBSgG2d6BfwcDDRDcFzEdjSVUJxtd): protocol: expected piece hash; context canceled; EOF Dec 27 13:43:37.323860349 (526h14m37.194943384s) [noise=true , address=45.91.116.10:4113 , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324005135 (526h14m37.1950882s) [noise=true , address=145.79.227.40:4079 , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RYKzn5zVSmxVbWwnBdX8oqtG1WE2MhPeEiWLf5dZeFAut6s2Y): protocol: expected piece hash; context canceled; EOF Dec 27 13:43:37.324022659 (526h14m37.195105704s) [noise=true , address=145.79.227.40:4079 , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324143378 (526h14m37.195226413s) [noise=true , address=77.90.31.16:4739 , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129B3N7yjJvwCzoVLTqWsPbcLQ5TQUyAzg1RCkbSAVKPWYTG8eb): protocol: expected piece hash; context canceled; EOF Dec 27 13:43:37.324154520 (526h14m37.195237555s) [noise=true , address=77.90.31.16:4739 , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324213442 (526h14m37.195296487s) [noise=true , address=64.49.13.169:4710 , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection Dec 27 13:43:37.324249360 (526h14m37.195332405s) [noise=true , address=64.49.13.169:4710 , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold Dec 27 13:43:37.324275360 (526h14m37.195358405s) [seg_pos={0 0}, upload=461595]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 13:43:37.324374018 (526h14m37.195457053s) [upload=461595]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:43:37.324381291 (526h14m37.195464337s) [upload=461595]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:43:37.679036406 (526h14m37.550119451s) [upload=461595]: Done uploading segments... Dec 27 13:43:37.679056985 (526h14m37.55014003s) [upload=461595]: Upload finished. err: <nil> Duration: 47.776907928s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [.XX.X.......XXXXX..XXXXX....XX....X.X.X.X.X..XXX.XX.XX.X.XX.] duration: 5.000948881s trace: https://jaeger.dev.storj.io/trace/2ec9e29774a07852 FAIL (2025-12-27 13:49:18 UTC) orbiter warning: context deadline exceeded Duration: 5.000976463s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..XX......X..X...X.XXX....X...X...X...XX.......X....X..XX.X.] duration: 5.876931846s trace: https://jaeger.dev.storj.io/trace/2e03edc8c63ef87e FAIL (2025-12-27 13:49:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 13:48:18.630263453 (526h19m18.501346488s) [upload=461661]: Starting upload Dec 27 13:48:18.630317655 (526h19m18.50140069s) [upload=461661]: Done starting upload Dec 27 13:48:18.630410092 (526h19m18.501493127s) [upload=461661]: Uploading segments... Dec 27 13:48:18.630503119 (526h19m18.501586164s) [upload=461661]: Got next segment. Inline: false Dec 27 13:48:18.630647574 (526h19m18.501730619s) [upload=461661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:48:18.630653545 (526h19m18.50173659s) [upload=461661]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:48:18.875983104 (526h19m18.747066149s) [seg_pos={0 0}, upload=461661]: Begin upload segment... Dec 27 13:48:18.876035624 (526h19m18.747118659s) [seg_pos={0 0}, upload=461661]: Done begin upload segment. Dec 27 13:48:18.876080330 (526h19m18.747163365s) [noise=true , address=145.79.227.5:4221 , piece_id=IHERUD5MB6ZXHT35XF2E36VTGMZQEXUPX2V6ZATML7H24IJHGW5A, seg_pos={0 0}, upload=461661]: Uploading piece... Dec 27 13:48:18.878349480 (526h19m18.749432515s) [noise=true , address=45.91.116.13:4154 , piece_id=6SBVBMUDW5WGNJHN4C3R2UCF4M4RN2WD66X2JAFPAWKV7BQJUEIA, seg_pos={0 0}, upload=461661]: Uploading piece... Dec 27 13:48:18.878940636 (526h19m18.750023671s) [noise=true , address=145.79.227.43:4158 , piece_id=G7PKXAYPICBQS3JLGENUDBZ2A67GG4V3YAJTULSAK47SIWSPCOIA, seg_pos={0 0}, upload=461661]: Uploading piece... Dec 27 13:48:18.879268781 (526h19m18.750351816s) [noise=true , address=45.91.116.12:4225 , piece_id=2X2JLREFT6QMWIZ2QVK7PTDSFRVWQSAPLDERZYY4LOGPCJDFX36Q, seg_pos={0 0}, upload=461661]: Uploading piece... Dec 27 13:48:18.879558162 (526h19m18.750641197s) [noise=true , address=77.90.31.15:4700 , piece_id=HH3WM3QPAUAB7DVPLOOXWGTAZUKWDHJ4FWMGGK57OMWHX3ALODHQ, seg_pos={0 0}, upload=461661]: Uploading piece... Dec 27 13:48:18.880675480 (526h19m18.751758525s) [noise=true , address=45.91.116.14:4189 , piece_id=WPU74TGTYRSND3YVFYNSGMKUHJLLHL5UOWCZRXRSTBXEGRAS2RIA, seg_pos={0 0}, upload=461661]: Uploading piece... Dec 27 13:48:18.881858342 (526h19m18.752941608s) [noise=true , address=145.79.227.10:4403 , piece_id=MWDCGGXBAV46ONF3HUW4G2EOPK42GUFIOYQIB6GYOFSYG5PJYU3A, seg_pos={0 0}, upload=461661]: Uploading piece... Dec 27 13:48:18.883113031 (526h19m18.754196076s) [noise=true , address=145.79.227.16:4911 , piece_id=36CWZSTS3HNYHNMGXRIVCTMXFQ7MNVKSFWEGUYUFIZUH4P3WY24A, seg_pos={0 0}, upload=461661]: Uploading piece... Dec 27 13:48:18.883956337 (526h19m18.755039382s) [noise=true , address=45.91.116.11:4120 , piece_id=IX6QG57CHO6YZ6BMGKNBKAHB7CQIEIBMVFIL4MYFJCUJDTCHH3WQ, seg_pos={0 0}, upload=461661]: Uploading piece... Dec 27 13:48:18.885569428 (526h19m18.756652463s) [noise=true , address=145.79.227.26:4655 , piece_id=LAEXQCDW56PNH6EQG7J5J34PHEYMYWTERWRRK3AXPS3SNNYFCERQ, seg_pos={0 0}, upload=461661]: Uploading piece... Dec 27 13:48:18.886115859 (526h19m18.757198894s) [noise=true , address=145.79.227.19:4078 , piece_id=UCU3ENF5XE6FJ2OMWPH3QP5LVK3JBTS6XJLGCNXZKQCPHVOC243A, seg_pos={0 0}, upload=461661]: Uploading piece... Dec 27 13:48:18.886725079 (526h19m18.757808114s) [noise=true , address=145.79.227.45:4231 , piece_id=NFURCAYX2ZSATN66DK6C6HYS55SQUGVYP64UV4RQ44LG55CCYEQQ, seg_pos={0 0}, upload=461661]: Uploading piece... Dec 27 13:48:18.888390570 (526h19m18.759473615s) [noise=true , address=145.79.227.49:4339 , piece_id=6WWZ475GEIXT2VDQ2PPFKPRQS3RMUFFDKDWOGH6Z3PDBVLCZW46Q, seg_pos={0 0}, upload=461661]: Uploading piece... Dec 27 13:48:18.889378662 (526h19m18.760461707s) [noise=true , address=145.79.227.35:4482 , piece_id=N3FIRID7URWTVFEVBPKIRJKCURJID6JHOH6AQCBRUOH32ARKHBDQ, seg_pos={0 0}, upload=461661]: Uploading piece... Dec 27 13:48:18.890481061 (526h19m18.761564096s) [noise=true , address=45.91.116.7:4009 , piece_id=FKHILBBIYHKCBAFW4Y7HEESQT22NNH6EUTRJ4QFTQJ7E767EJ22A, seg_pos={0 0}, upload=461661]: Uploading piece... Dec 27 13:48:18.891020067 (526h19m18.762103102s) [noise=true , address=145.79.227.46:4602 , piece_id=REJXSCMMAHBISLHLSX7JTIHLYYT5VK5XQPYY76HZYORHHVIWOQTA, seg_pos={0 0}, upload=461661]: Uploading piece... Dec 27 13:48:20.550244355 (526h19m20.42132739s) [upload=461661]: Next returned nil segment Dec 27 13:48:20.550258803 (526h19m20.421341848s) [upload=461661]: Waiting for error group managing segments... Dec 27 13:48:20.896584165 (526h19m20.76766721s) [noise=true , address=145.79.227.10:4403 , piece_id=MWDCGGXBAV46ONF3HUW4G2EOPK42GUFIOYQIB6GYOFSYG5PJYU3A, seg_pos={0 0}, upload=461661]: Done uploading piece. err: <nil> Dec 27 13:48:20.900628606 (526h19m20.771711652s) [noise=true , address=145.79.227.45:4231 , piece_id=NFURCAYX2ZSATN66DK6C6HYS55SQUGVYP64UV4RQ44LG55CCYEQQ, seg_pos={0 0}, upload=461661]: Done uploading piece. err: <nil> Dec 27 13:48:20.909473388 (526h19m20.780556423s) [noise=true , address=145.79.227.35:4482 , piece_id=N3FIRID7URWTVFEVBPKIRJKCURJID6JHOH6AQCBRUOH32ARKHBDQ, seg_pos={0 0}, upload=461661]: Done uploading piece. err: <nil> Dec 27 13:48:20.916144421 (526h19m20.787227466s) [noise=true , address=145.79.227.49:4339 , piece_id=6WWZ475GEIXT2VDQ2PPFKPRQS3RMUFFDKDWOGH6Z3PDBVLCZW46Q, seg_pos={0 0}, upload=461661]: Done uploading piece. err: <nil> Dec 27 13:48:20.921735728 (526h19m20.792818763s) [noise=true , address=145.79.227.19:4078 , piece_id=UCU3ENF5XE6FJ2OMWPH3QP5LVK3JBTS6XJLGCNXZKQCPHVOC243A, seg_pos={0 0}, upload=461661]: Done uploading piece. err: <nil> Dec 27 13:48:20.922393290 (526h19m20.793476325s) [noise=true , address=145.79.227.46:4602 , piece_id=REJXSCMMAHBISLHLSX7JTIHLYYT5VK5XQPYY76HZYORHHVIWOQTA, seg_pos={0 0}, upload=461661]: Done uploading piece. err: <nil> Dec 27 13:48:20.928719496 (526h19m20.799802531s) [noise=true , address=145.79.227.26:4655 , piece_id=LAEXQCDW56PNH6EQG7J5J34PHEYMYWTERWRRK3AXPS3SNNYFCERQ, seg_pos={0 0}, upload=461661]: Done uploading piece. err: <nil> Dec 27 13:48:20.929064664 (526h19m20.800147709s) [noise=true , address=145.79.227.16:4911 , piece_id=36CWZSTS3HNYHNMGXRIVCTMXFQ7MNVKSFWEGUYUFIZUH4P3WY24A, seg_pos={0 0}, upload=461661]: Done uploading piece. err: <nil> Dec 27 13:48:20.930809666 (526h19m20.801892701s) [noise=true , address=145.79.227.5:4221 , piece_id=IHERUD5MB6ZXHT35XF2E36VTGMZQEXUPX2V6ZATML7H24IJHGW5A, seg_pos={0 0}, upload=461661]: Done uploading piece. err: <nil> Dec 27 13:48:21.074684198 (526h19m20.945767243s) [noise=true , address=77.90.31.15:4700 , piece_id=HH3WM3QPAUAB7DVPLOOXWGTAZUKWDHJ4FWMGGK57OMWHX3ALODHQ, seg_pos={0 0}, upload=461661]: Done uploading piece. err: <nil> Dec 27 13:48:21.246902702 (526h19m21.117985737s) [noise=true , address=45.91.116.13:4154 , piece_id=6SBVBMUDW5WGNJHN4C3R2UCF4M4RN2WD66X2JAFPAWKV7BQJUEIA, seg_pos={0 0}, upload=461661]: Done uploading piece. err: <nil> Dec 27 13:48:21.269436540 (526h19m21.140519575s) [noise=true , address=45.91.116.14:4189 , piece_id=WPU74TGTYRSND3YVFYNSGMKUHJLLHL5UOWCZRXRSTBXEGRAS2RIA, seg_pos={0 0}, upload=461661]: Done uploading piece. err: <nil> Dec 27 13:48:21.357514185 (526h19m21.22859723s) [noise=true , address=45.91.116.12:4225 , piece_id=2X2JLREFT6QMWIZ2QVK7PTDSFRVWQSAPLDERZYY4LOGPCJDFX36Q, seg_pos={0 0}, upload=461661]: Done uploading piece. err: <nil> Dec 27 13:48:21.357544753 (526h19m21.228627798s) [seg_pos={0 0}, upload=461661]: Segment reached optimal threshold of 13 pieces. Dec 27 13:48:21.358103507 (526h19m21.229186572s) [noise=true , address=45.91.116.11:4120 , piece_id=IX6QG57CHO6YZ6BMGKNBKAHB7CQIEIBMVFIL4MYFJCUJDTCHH3WQ, seg_pos={0 0}, upload=461661]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 27 13:48:21.358129096 (526h19m21.229212161s) [noise=true , address=45.91.116.11:4120 , piece_id=IX6QG57CHO6YZ6BMGKNBKAHB7CQIEIBMVFIL4MYFJCUJDTCHH3WQ, seg_pos={0 0}, upload=461661]: Long tail cancellation due to optimal threshold Dec 27 13:48:21.358166747 (526h19m21.229249873s) [noise=true , address=145.79.227.43:4158 , piece_id=G7PKXAYPICBQS3JLGENUDBZ2A67GG4V3YAJTULSAK47SIWSPCOIA, seg_pos={0 0}, upload=461661]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 27 13:48:21.358177638 (526h19m21.229260673s) [noise=true , address=145.79.227.43:4158 , piece_id=G7PKXAYPICBQS3JLGENUDBZ2A67GG4V3YAJTULSAK47SIWSPCOIA, seg_pos={0 0}, upload=461661]: Long tail cancellation due to optimal threshold Dec 27 13:48:21.358415261 (526h19m21.229498296s) [noise=true , address=45.91.116.7:4009 , piece_id=FKHILBBIYHKCBAFW4Y7HEESQT22NNH6EUTRJ4QFTQJ7E767EJ22A, seg_pos={0 0}, upload=461661]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: context canceled; context canceled Dec 27 13:48:21.358422775 (526h19m21.22950581s) [noise=true , address=45.91.116.7:4009 , piece_id=FKHILBBIYHKCBAFW4Y7HEESQT22NNH6EUTRJ4QFTQJ7E767EJ22A, seg_pos={0 0}, upload=461661]: Long tail cancellation due to optimal threshold Dec 27 13:48:21.358440319 (526h19m21.229523354s) [seg_pos={0 0}, upload=461661]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:48:21.358538165 (526h19m21.22962121s) [upload=461661]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:48:21.358543255 (526h19m21.22962631s) [upload=461661]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:48:21.648807477 (526h19m21.519890512s) [upload=461661]: Done uploading segments... Dec 27 13:48:21.648827004 (526h19m21.519910039s) [upload=461661]: Upload finished. err: <nil> Duration: 5.876963796s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)