Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 864157 reports (Page 11707 of 17284)

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...X............X.X.XX.XX.XX..X.....XX...X.X.X.X....XX......] duration: 20.377739702s trace: https://jaeger.dev.storj.io/trace/34dcbf1241a033f7 FAIL (2025-12-08 15:50:13 UTC) orbiter warning: context deadline exceeded Duration: 20.377768335s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [XXXXX.XXX..X....X...X...X.X.X.....XX.X.X.XXXX.X...XX.X..XX.X] duration: 5.001393152s trace: https://jaeger.dev.storj.io/trace/767f6b942710b5f FAIL (2025-12-08 15:50:07 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001422858s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.437741095s trace: https://jaeger.dev.storj.io/trace/4195524620a3e2c2 FAIL (2025-12-08 15:50:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments... Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces. Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments... Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.437775882s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [....................X.........................X..X...X......] duration: 20.53769101s trace: https://jaeger.dev.storj.io/trace/33a3f0149193afa6 FAIL (2025-12-08 15:50:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 20.537719824s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 7.889623604s trace: https://jaeger.dev.storj.io/trace/153706e03a29fd88 FAIL (2025-12-08 15:50:01 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 7.889656956s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [.XXXXX.XXX..X....X...X...X.X.X.....XX.X.X.XXXX.X...XX.X..XX.] duration: 3.41616169s trace: https://jaeger.dev.storj.io/trace/330fd88e71df9e64 FAIL (2025-12-08 15:49:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.416184042s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 7.074709972s trace: https://jaeger.dev.storj.io/trace/54ba7b1e9d79a182 FAIL (2025-12-08 15:49:30 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 7.074738066s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.....................X.........................X..X...X.....] duration: 19.446842446s trace: https://jaeger.dev.storj.io/trace/33a3f0149193afa6 FAIL (2025-12-08 15:49:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.44687056s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................X.........X...X..............X.........] duration: 7.577930855s trace: https://jaeger.dev.storj.io/trace/3711a2689ed652fd FAIL (2025-12-08 15:49:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: VKXVHWX7LFKBH4E32F6CMAHMQX2P5AREIEIWHTXK7U22TEWDMRFQ) context deadline exceeded; piecestore: (Node ID: 18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, Piece ID: AUU7NTJQWR52XFWHVONDFHZHKXPOUS2W4UL4OF4XGTWV4ABETSHQ) context deadline exceeded; piecestore: (Node ID: 12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, Piece ID: 2AI2DCS6TNPJGURTBMNDKT5N3GISWDVOC5VTLIP6MVUAEWP2V3ZQ) context deadline exceeded ======LOGS====== Dec 8 15:43:33.737263649 (72h14m33.608346694s) [upload=63248]: Starting upload Dec 8 15:43:33.737410899 (72h14m33.608493944s) [upload=63248]: Done starting upload Dec 8 15:43:33.737492094 (72h14m33.608575149s) [upload=63248]: Uploading segments... Dec 8 15:43:33.739621138 (72h14m33.610704183s) [upload=63248]: Got next segment. Inline: false Dec 8 15:43:33.739675270 (72h14m33.610758305s) [upload=63248]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 15:43:33.739681903 (72h14m33.610764938s) [upload=63248]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 15:43:33.988217234 (72h14m33.859300269s) [seg_pos={0 0}, upload=63248]: Begin upload segment... Dec 8 15:43:33.988366548 (72h14m33.859449583s) [seg_pos={0 0}, upload=63248]: Done begin upload segment. Dec 8 15:43:33.988395153 (72h14m33.859478198s) [noise=true , address=77.90.31.19:4842 , piece_id=D6RHPHJCJTV7I6UF5HQSPFFNVVHJDYXOIY3C747PLFFACW32INWA, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.988434698 (72h14m33.859517763s) [noise=true , address=145.79.227.27:4689 , piece_id=LAMO54F6RV2YAJQNHCIOJNSAQGCXK23WHVJJ6VC2REYJ7LUW5R3Q, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.989099694 (72h14m33.860182729s) [noise=true , address=145.79.227.29:4943 , piece_id=TJFW4JKEHHCDTMPVS46BI4OYF35BVHE3VIFM6LD2CJKRFCZXL5FA, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.989662838 (72h14m33.860745873s) [noise=true , address=145.79.227.22:4510 , piece_id=BVTVWVC7VRYL7TREJ63BRWY6LAHU2N5AHYJR3JRHQPPEEITTMFVQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.990143793 (72h14m33.861226828s) [noise=true , address=145.79.227.43:4157 , piece_id=3P7NOOCSNAUBKH4SVL2SUSQOADSA3IKRHTPRL4P7VGH62DL23LFQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.990589972 (72h14m33.861673007s) [noise=true , address=77.90.31.10:4519 , piece_id=PGLC7MAH7IJ7GP6DTN65WUESDI3Y2JNDRVFR3QJJBNZV2H2SFJSA, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.991038847 (72h14m33.862121882s) [noise=true , address=145.79.227.39:4015 , piece_id=AUU7NTJQWR52XFWHVONDFHZHKXPOUS2W4UL4OF4XGTWV4ABETSHQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.991412218 (72h14m33.862495263s) [noise=true , address=145.79.227.38:4978 , piece_id=2AI2DCS6TNPJGURTBMNDKT5N3GISWDVOC5VTLIP6MVUAEWP2V3ZQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.991728711 (72h14m33.862811746s) [noise=true , address=145.79.227.38:4979 , piece_id=4A4TZ2YT5KBQVYX7XSKIUMFJBLR3R7NJOZCVXEVVIK67VC744VXA, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.992474100 (72h14m33.863557145s) [noise=true , address=145.79.227.21:4475 , piece_id=PLY2YAALNNQKXXF5MZUUKDUV4HP7XUDRD3QXJTTKQI6FCGLHEA7A, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.992900823 (72h14m33.863983858s) [noise=true , address=145.79.227.28:4727 , piece_id=BSP57B4M3L7UIEDLVM2ASNQRK5BYLGTMAJ633QN4WFNS3OCNWDGQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.993301476 (72h14m33.864384521s) [noise=true , address=145.79.227.41:4085 , piece_id=ODJABK4R6SHNXNNQ47WI7W4YB4NR5A6QDTUE4WXVUFBVE7X3AJCQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.993926856 (72h14m33.865009922s) [noise=true , address=145.79.227.25:4619 , piece_id=CJJ7CONWO42FFSV7QMB4JSATW7BUDIS6KJXYX2SIBQMWE3UTYA4A, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.994260803 (72h14m33.865343838s) [noise=true , address=77.90.31.14:4661 , piece_id=BON7N46ZON2BYBD4UHNHPAECUF2OL7KV6B7Z2SZWVS6OGTEAK3RA, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.994756687 (72h14m33.865839732s) [noise=true , address=145.79.227.31:4799 , piece_id=VKXVHWX7LFKBH4E32F6CMAHMQX2P5AREIEIWHTXK7U22TEWDMRFQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.995123304 (72h14m33.866206349s) [noise=true , address=145.79.227.26:4653 , piece_id=LMBLPSOTMKPFS4KGAJOSRR2EM2MVKJIXSRE5MIB4SF552S2O3QYQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:35.816987512 (72h14m35.688070547s) [upload=63248]: Next returned nil segment Dec 8 15:43:35.817000286 (72h14m35.688083331s) [upload=63248]: Waiting for error group managing segments... Dec 8 15:43:36.138037836 (72h14m36.009120871s) [noise=true , address=77.90.31.10:4519 , piece_id=PGLC7MAH7IJ7GP6DTN65WUESDI3Y2JNDRVFR3QJJBNZV2H2SFJSA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.161877783 (72h14m36.032960818s) [noise=true , address=145.79.227.38:4979 , piece_id=4A4TZ2YT5KBQVYX7XSKIUMFJBLR3R7NJOZCVXEVVIK67VC744VXA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.164129480 (72h14m36.035212525s) [noise=true , address=145.79.227.31:4799 , piece_id=VKXVHWX7LFKBH4E32F6CMAHMQX2P5AREIEIWHTXK7U22TEWDMRFQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.166039237 (72h14m36.037122272s) [noise=true , address=145.79.227.41:4085 , piece_id=ODJABK4R6SHNXNNQ47WI7W4YB4NR5A6QDTUE4WXVUFBVE7X3AJCQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.166480136 (72h14m36.037563181s) [noise=true , address=145.79.227.25:4619 , piece_id=CJJ7CONWO42FFSV7QMB4JSATW7BUDIS6KJXYX2SIBQMWE3UTYA4A, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.167642099 (72h14m36.038725134s) [noise=true , address=145.79.227.43:4157 , piece_id=3P7NOOCSNAUBKH4SVL2SUSQOADSA3IKRHTPRL4P7VGH62DL23LFQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.170345487 (72h14m36.041428532s) [noise=true , address=77.90.31.19:4842 , piece_id=D6RHPHJCJTV7I6UF5HQSPFFNVVHJDYXOIY3C747PLFFACW32INWA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.173595836 (72h14m36.044678881s) [noise=true , address=145.79.227.21:4475 , piece_id=PLY2YAALNNQKXXF5MZUUKDUV4HP7XUDRD3QXJTTKQI6FCGLHEA7A, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.185245680 (72h14m36.056328725s) [noise=true , address=145.79.227.28:4727 , piece_id=BSP57B4M3L7UIEDLVM2ASNQRK5BYLGTMAJ633QN4WFNS3OCNWDGQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.188436325 (72h14m36.05951937s) [noise=true , address=77.90.31.14:4661 , piece_id=BON7N46ZON2BYBD4UHNHPAECUF2OL7KV6B7Z2SZWVS6OGTEAK3RA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.199146960 (72h14m36.070229995s) [noise=true , address=145.79.227.38:4978 , piece_id=2AI2DCS6TNPJGURTBMNDKT5N3GISWDVOC5VTLIP6MVUAEWP2V3ZQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.484809459 (72h14m36.355892504s) [noise=true , address=145.79.227.39:4015 , piece_id=AUU7NTJQWR52XFWHVONDFHZHKXPOUS2W4UL4OF4XGTWV4ABETSHQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:37.179185330 (72h14m37.050268365s) [noise=true , address=145.79.227.22:4510 , piece_id=BVTVWVC7VRYL7TREJ63BRWY6LAHU2N5AHYJR3JRHQPPEEITTMFVQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:37.179203285 (72h14m37.05028633s) [seg_pos={0 0}, upload=63248]: Segment reached optimal threshold of 13 pieces. Dec 8 15:43:37.179369642 (72h14m37.050452687s) [noise=true , address=145.79.227.26:4653 , piece_id=LMBLPSOTMKPFS4KGAJOSRR2EM2MVKJIXSRE5MIB4SF552S2O3QYQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: expected piece hash; context canceled; EOF Dec 8 15:43:37.179375573 (72h14m37.050458608s) [noise=true , address=145.79.227.26:4653 , piece_id=LMBLPSOTMKPFS4KGAJOSRR2EM2MVKJIXSRE5MIB4SF552S2O3QYQ, seg_pos={0 0}, upload=63248]: Long tail cancellation due to optimal threshold Dec 8 15:43:37.179483609 (72h14m37.050566654s) [noise=true , address=145.79.227.29:4943 , piece_id=TJFW4JKEHHCDTMPVS46BI4OYF35BVHE3VIFM6LD2CJKRFCZXL5FA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 8 15:43:37.179488698 (72h14m37.050571743s) [noise=true , address=145.79.227.29:4943 , piece_id=TJFW4JKEHHCDTMPVS46BI4OYF35BVHE3VIFM6LD2CJKRFCZXL5FA, seg_pos={0 0}, upload=63248]: Long tail cancellation due to optimal threshold Dec 8 15:43:37.179679632 (72h14m37.050762687s) [noise=true , address=145.79.227.27:4689 , piece_id=LAMO54F6RV2YAJQNHCIOJNSAQGCXK23WHVJJ6VC2REYJ7LUW5R3Q, seg_pos={0 0}, upload=63248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ): protocol: write tcp 188.166.225.170:57686->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:57686->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57686->145.79.227.27:4689: use of closed network connection Dec 8 15:43:37.179722844 (72h14m37.050805879s) [noise=true , address=145.79.227.27:4689 , piece_id=LAMO54F6RV2YAJQNHCIOJNSAQGCXK23WHVJJ6VC2REYJ7LUW5R3Q, seg_pos={0 0}, upload=63248]: Long tail cancellation due to optimal threshold Dec 8 15:43:37.179754174 (72h14m37.050837209s) [seg_pos={0 0}, upload=63248]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 15:43:37.179977008 (72h14m37.051060043s) [upload=63248]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 15:43:37.179985915 (72h14m37.05106895s) [upload=63248]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 15:43:37.492140220 (72h14m37.363223265s) [upload=63248]: Done uploading segments... Dec 8 15:43:37.492158695 (72h14m37.36324173s) [upload=63248]: Upload finished. err: <nil> Duration: 7.577965151s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 7.528305179s trace: https://jaeger.dev.storj.io/trace/153706e03a29fd88 FAIL (2025-12-08 15:49:27 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 7.528329194s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.401821577s trace: https://jaeger.dev.storj.io/trace/54ba7b1e9d79a182 FAIL (2025-12-08 15:49:15 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.40184958s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [X.XXXXX.XXX..X....X...X...X.X.X.....XX.X.X.XXXX.X...XX.X..XX] duration: 5.001115703s trace: https://jaeger.dev.storj.io/trace/330fd88e71df9e64 FAIL (2025-12-08 15:49:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001133066s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 6.397728806s trace: https://jaeger.dev.storj.io/trace/4195524620a3e2c2 FAIL (2025-12-08 15:49:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments... Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces. Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments... Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.397760487s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [....X............X.X.XX.XX.XX..X.....XX...X.X.X.X....XX.....] duration: 8.219739655s trace: https://jaeger.dev.storj.io/trace/34dcbf1241a033f7 FAIL (2025-12-08 15:48:52 UTC) orbiter warning: context deadline exceeded Duration: 8.219762298s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [.X.XXXXX.XXX..X....X...X...X.X.X.....XX.X.X.XXXX.X...XX.X..X] duration: 5.00113486s trace: https://jaeger.dev.storj.io/trace/4d9485713cf2a5c5 FAIL (2025-12-08 15:48:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001156652s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 7.484922445s trace: https://jaeger.dev.storj.io/trace/153706e03a29fd88 FAIL (2025-12-08 15:48:37 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 7.484949887s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 6.10599839s trace: https://jaeger.dev.storj.io/trace/4195524620a3e2c2 FAIL (2025-12-08 15:48:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments... Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces. Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments... Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.106018658s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.730847639s trace: https://jaeger.dev.storj.io/trace/54ba7b1e9d79a182 FAIL (2025-12-08 15:48:23 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.730896894s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................X.........X...X..............X........] duration: 8.150789323s trace: https://jaeger.dev.storj.io/trace/3711a2689ed652fd FAIL (2025-12-08 15:48:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: VKXVHWX7LFKBH4E32F6CMAHMQX2P5AREIEIWHTXK7U22TEWDMRFQ) context deadline exceeded; piecestore: (Node ID: 18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, Piece ID: AUU7NTJQWR52XFWHVONDFHZHKXPOUS2W4UL4OF4XGTWV4ABETSHQ) context deadline exceeded; piecestore: (Node ID: 12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, Piece ID: 2AI2DCS6TNPJGURTBMNDKT5N3GISWDVOC5VTLIP6MVUAEWP2V3ZQ) context deadline exceeded ======LOGS====== Dec 8 15:43:33.737263649 (72h14m33.608346694s) [upload=63248]: Starting upload Dec 8 15:43:33.737410899 (72h14m33.608493944s) [upload=63248]: Done starting upload Dec 8 15:43:33.737492094 (72h14m33.608575149s) [upload=63248]: Uploading segments... Dec 8 15:43:33.739621138 (72h14m33.610704183s) [upload=63248]: Got next segment. Inline: false Dec 8 15:43:33.739675270 (72h14m33.610758305s) [upload=63248]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 15:43:33.739681903 (72h14m33.610764938s) [upload=63248]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 15:43:33.988217234 (72h14m33.859300269s) [seg_pos={0 0}, upload=63248]: Begin upload segment... Dec 8 15:43:33.988366548 (72h14m33.859449583s) [seg_pos={0 0}, upload=63248]: Done begin upload segment. Dec 8 15:43:33.988395153 (72h14m33.859478198s) [noise=true , address=77.90.31.19:4842 , piece_id=D6RHPHJCJTV7I6UF5HQSPFFNVVHJDYXOIY3C747PLFFACW32INWA, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.988434698 (72h14m33.859517763s) [noise=true , address=145.79.227.27:4689 , piece_id=LAMO54F6RV2YAJQNHCIOJNSAQGCXK23WHVJJ6VC2REYJ7LUW5R3Q, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.989099694 (72h14m33.860182729s) [noise=true , address=145.79.227.29:4943 , piece_id=TJFW4JKEHHCDTMPVS46BI4OYF35BVHE3VIFM6LD2CJKRFCZXL5FA, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.989662838 (72h14m33.860745873s) [noise=true , address=145.79.227.22:4510 , piece_id=BVTVWVC7VRYL7TREJ63BRWY6LAHU2N5AHYJR3JRHQPPEEITTMFVQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.990143793 (72h14m33.861226828s) [noise=true , address=145.79.227.43:4157 , piece_id=3P7NOOCSNAUBKH4SVL2SUSQOADSA3IKRHTPRL4P7VGH62DL23LFQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.990589972 (72h14m33.861673007s) [noise=true , address=77.90.31.10:4519 , piece_id=PGLC7MAH7IJ7GP6DTN65WUESDI3Y2JNDRVFR3QJJBNZV2H2SFJSA, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.991038847 (72h14m33.862121882s) [noise=true , address=145.79.227.39:4015 , piece_id=AUU7NTJQWR52XFWHVONDFHZHKXPOUS2W4UL4OF4XGTWV4ABETSHQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.991412218 (72h14m33.862495263s) [noise=true , address=145.79.227.38:4978 , piece_id=2AI2DCS6TNPJGURTBMNDKT5N3GISWDVOC5VTLIP6MVUAEWP2V3ZQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.991728711 (72h14m33.862811746s) [noise=true , address=145.79.227.38:4979 , piece_id=4A4TZ2YT5KBQVYX7XSKIUMFJBLR3R7NJOZCVXEVVIK67VC744VXA, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.992474100 (72h14m33.863557145s) [noise=true , address=145.79.227.21:4475 , piece_id=PLY2YAALNNQKXXF5MZUUKDUV4HP7XUDRD3QXJTTKQI6FCGLHEA7A, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.992900823 (72h14m33.863983858s) [noise=true , address=145.79.227.28:4727 , piece_id=BSP57B4M3L7UIEDLVM2ASNQRK5BYLGTMAJ633QN4WFNS3OCNWDGQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.993301476 (72h14m33.864384521s) [noise=true , address=145.79.227.41:4085 , piece_id=ODJABK4R6SHNXNNQ47WI7W4YB4NR5A6QDTUE4WXVUFBVE7X3AJCQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.993926856 (72h14m33.865009922s) [noise=true , address=145.79.227.25:4619 , piece_id=CJJ7CONWO42FFSV7QMB4JSATW7BUDIS6KJXYX2SIBQMWE3UTYA4A, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.994260803 (72h14m33.865343838s) [noise=true , address=77.90.31.14:4661 , piece_id=BON7N46ZON2BYBD4UHNHPAECUF2OL7KV6B7Z2SZWVS6OGTEAK3RA, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.994756687 (72h14m33.865839732s) [noise=true , address=145.79.227.31:4799 , piece_id=VKXVHWX7LFKBH4E32F6CMAHMQX2P5AREIEIWHTXK7U22TEWDMRFQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.995123304 (72h14m33.866206349s) [noise=true , address=145.79.227.26:4653 , piece_id=LMBLPSOTMKPFS4KGAJOSRR2EM2MVKJIXSRE5MIB4SF552S2O3QYQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:35.816987512 (72h14m35.688070547s) [upload=63248]: Next returned nil segment Dec 8 15:43:35.817000286 (72h14m35.688083331s) [upload=63248]: Waiting for error group managing segments... Dec 8 15:43:36.138037836 (72h14m36.009120871s) [noise=true , address=77.90.31.10:4519 , piece_id=PGLC7MAH7IJ7GP6DTN65WUESDI3Y2JNDRVFR3QJJBNZV2H2SFJSA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.161877783 (72h14m36.032960818s) [noise=true , address=145.79.227.38:4979 , piece_id=4A4TZ2YT5KBQVYX7XSKIUMFJBLR3R7NJOZCVXEVVIK67VC744VXA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.164129480 (72h14m36.035212525s) [noise=true , address=145.79.227.31:4799 , piece_id=VKXVHWX7LFKBH4E32F6CMAHMQX2P5AREIEIWHTXK7U22TEWDMRFQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.166039237 (72h14m36.037122272s) [noise=true , address=145.79.227.41:4085 , piece_id=ODJABK4R6SHNXNNQ47WI7W4YB4NR5A6QDTUE4WXVUFBVE7X3AJCQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.166480136 (72h14m36.037563181s) [noise=true , address=145.79.227.25:4619 , piece_id=CJJ7CONWO42FFSV7QMB4JSATW7BUDIS6KJXYX2SIBQMWE3UTYA4A, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.167642099 (72h14m36.038725134s) [noise=true , address=145.79.227.43:4157 , piece_id=3P7NOOCSNAUBKH4SVL2SUSQOADSA3IKRHTPRL4P7VGH62DL23LFQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.170345487 (72h14m36.041428532s) [noise=true , address=77.90.31.19:4842 , piece_id=D6RHPHJCJTV7I6UF5HQSPFFNVVHJDYXOIY3C747PLFFACW32INWA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.173595836 (72h14m36.044678881s) [noise=true , address=145.79.227.21:4475 , piece_id=PLY2YAALNNQKXXF5MZUUKDUV4HP7XUDRD3QXJTTKQI6FCGLHEA7A, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.185245680 (72h14m36.056328725s) [noise=true , address=145.79.227.28:4727 , piece_id=BSP57B4M3L7UIEDLVM2ASNQRK5BYLGTMAJ633QN4WFNS3OCNWDGQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.188436325 (72h14m36.05951937s) [noise=true , address=77.90.31.14:4661 , piece_id=BON7N46ZON2BYBD4UHNHPAECUF2OL7KV6B7Z2SZWVS6OGTEAK3RA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.199146960 (72h14m36.070229995s) [noise=true , address=145.79.227.38:4978 , piece_id=2AI2DCS6TNPJGURTBMNDKT5N3GISWDVOC5VTLIP6MVUAEWP2V3ZQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.484809459 (72h14m36.355892504s) [noise=true , address=145.79.227.39:4015 , piece_id=AUU7NTJQWR52XFWHVONDFHZHKXPOUS2W4UL4OF4XGTWV4ABETSHQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:37.179185330 (72h14m37.050268365s) [noise=true , address=145.79.227.22:4510 , piece_id=BVTVWVC7VRYL7TREJ63BRWY6LAHU2N5AHYJR3JRHQPPEEITTMFVQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:37.179203285 (72h14m37.05028633s) [seg_pos={0 0}, upload=63248]: Segment reached optimal threshold of 13 pieces. Dec 8 15:43:37.179369642 (72h14m37.050452687s) [noise=true , address=145.79.227.26:4653 , piece_id=LMBLPSOTMKPFS4KGAJOSRR2EM2MVKJIXSRE5MIB4SF552S2O3QYQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: expected piece hash; context canceled; EOF Dec 8 15:43:37.179375573 (72h14m37.050458608s) [noise=true , address=145.79.227.26:4653 , piece_id=LMBLPSOTMKPFS4KGAJOSRR2EM2MVKJIXSRE5MIB4SF552S2O3QYQ, seg_pos={0 0}, upload=63248]: Long tail cancellation due to optimal threshold Dec 8 15:43:37.179483609 (72h14m37.050566654s) [noise=true , address=145.79.227.29:4943 , piece_id=TJFW4JKEHHCDTMPVS46BI4OYF35BVHE3VIFM6LD2CJKRFCZXL5FA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 8 15:43:37.179488698 (72h14m37.050571743s) [noise=true , address=145.79.227.29:4943 , piece_id=TJFW4JKEHHCDTMPVS46BI4OYF35BVHE3VIFM6LD2CJKRFCZXL5FA, seg_pos={0 0}, upload=63248]: Long tail cancellation due to optimal threshold Dec 8 15:43:37.179679632 (72h14m37.050762687s) [noise=true , address=145.79.227.27:4689 , piece_id=LAMO54F6RV2YAJQNHCIOJNSAQGCXK23WHVJJ6VC2REYJ7LUW5R3Q, seg_pos={0 0}, upload=63248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ): protocol: write tcp 188.166.225.170:57686->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:57686->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57686->145.79.227.27:4689: use of closed network connection Dec 8 15:43:37.179722844 (72h14m37.050805879s) [noise=true , address=145.79.227.27:4689 , piece_id=LAMO54F6RV2YAJQNHCIOJNSAQGCXK23WHVJJ6VC2REYJ7LUW5R3Q, seg_pos={0 0}, upload=63248]: Long tail cancellation due to optimal threshold Dec 8 15:43:37.179754174 (72h14m37.050837209s) [seg_pos={0 0}, upload=63248]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 15:43:37.179977008 (72h14m37.051060043s) [upload=63248]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 15:43:37.179985915 (72h14m37.05106895s) [upload=63248]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 15:43:37.492140220 (72h14m37.363223265s) [upload=63248]: Done uploading segments... Dec 8 15:43:37.492158695 (72h14m37.36324173s) [upload=63248]: Upload finished. err: <nil> Duration: 8.150827947s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.089304753s trace: https://jaeger.dev.storj.io/trace/153706e03a29fd88 FAIL (2025-12-08 15:48:14 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 6.089333698s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [......................X.........................X..X...X....] duration: 7.130639924s trace: https://jaeger.dev.storj.io/trace/33a3f0149193afa6 FAIL (2025-12-08 15:48:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.130667988s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 7.426748604s trace: https://jaeger.dev.storj.io/trace/54ba7b1e9d79a182 FAIL (2025-12-08 15:48:04 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 7.426781487s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.....X............X.X.XX.XX.XX..X.....XX...X.X.X.X....XX....] duration: 18.706947169s trace: https://jaeger.dev.storj.io/trace/34dcbf1241a033f7 FAIL (2025-12-08 15:47:57 UTC) orbiter warning: context deadline exceeded Duration: 18.706978127s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................X.........X...X..............X.......] duration: 7.224680386s trace: https://jaeger.dev.storj.io/trace/3711a2689ed652fd FAIL (2025-12-08 15:47:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: VKXVHWX7LFKBH4E32F6CMAHMQX2P5AREIEIWHTXK7U22TEWDMRFQ) context deadline exceeded; piecestore: (Node ID: 18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, Piece ID: AUU7NTJQWR52XFWHVONDFHZHKXPOUS2W4UL4OF4XGTWV4ABETSHQ) context deadline exceeded; piecestore: (Node ID: 12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, Piece ID: 2AI2DCS6TNPJGURTBMNDKT5N3GISWDVOC5VTLIP6MVUAEWP2V3ZQ) context deadline exceeded ======LOGS====== Dec 8 15:43:33.737263649 (72h14m33.608346694s) [upload=63248]: Starting upload Dec 8 15:43:33.737410899 (72h14m33.608493944s) [upload=63248]: Done starting upload Dec 8 15:43:33.737492094 (72h14m33.608575149s) [upload=63248]: Uploading segments... Dec 8 15:43:33.739621138 (72h14m33.610704183s) [upload=63248]: Got next segment. Inline: false Dec 8 15:43:33.739675270 (72h14m33.610758305s) [upload=63248]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 15:43:33.739681903 (72h14m33.610764938s) [upload=63248]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 15:43:33.988217234 (72h14m33.859300269s) [seg_pos={0 0}, upload=63248]: Begin upload segment... Dec 8 15:43:33.988366548 (72h14m33.859449583s) [seg_pos={0 0}, upload=63248]: Done begin upload segment. Dec 8 15:43:33.988395153 (72h14m33.859478198s) [noise=true , address=77.90.31.19:4842 , piece_id=D6RHPHJCJTV7I6UF5HQSPFFNVVHJDYXOIY3C747PLFFACW32INWA, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.988434698 (72h14m33.859517763s) [noise=true , address=145.79.227.27:4689 , piece_id=LAMO54F6RV2YAJQNHCIOJNSAQGCXK23WHVJJ6VC2REYJ7LUW5R3Q, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.989099694 (72h14m33.860182729s) [noise=true , address=145.79.227.29:4943 , piece_id=TJFW4JKEHHCDTMPVS46BI4OYF35BVHE3VIFM6LD2CJKRFCZXL5FA, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.989662838 (72h14m33.860745873s) [noise=true , address=145.79.227.22:4510 , piece_id=BVTVWVC7VRYL7TREJ63BRWY6LAHU2N5AHYJR3JRHQPPEEITTMFVQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.990143793 (72h14m33.861226828s) [noise=true , address=145.79.227.43:4157 , piece_id=3P7NOOCSNAUBKH4SVL2SUSQOADSA3IKRHTPRL4P7VGH62DL23LFQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.990589972 (72h14m33.861673007s) [noise=true , address=77.90.31.10:4519 , piece_id=PGLC7MAH7IJ7GP6DTN65WUESDI3Y2JNDRVFR3QJJBNZV2H2SFJSA, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.991038847 (72h14m33.862121882s) [noise=true , address=145.79.227.39:4015 , piece_id=AUU7NTJQWR52XFWHVONDFHZHKXPOUS2W4UL4OF4XGTWV4ABETSHQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.991412218 (72h14m33.862495263s) [noise=true , address=145.79.227.38:4978 , piece_id=2AI2DCS6TNPJGURTBMNDKT5N3GISWDVOC5VTLIP6MVUAEWP2V3ZQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.991728711 (72h14m33.862811746s) [noise=true , address=145.79.227.38:4979 , piece_id=4A4TZ2YT5KBQVYX7XSKIUMFJBLR3R7NJOZCVXEVVIK67VC744VXA, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.992474100 (72h14m33.863557145s) [noise=true , address=145.79.227.21:4475 , piece_id=PLY2YAALNNQKXXF5MZUUKDUV4HP7XUDRD3QXJTTKQI6FCGLHEA7A, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.992900823 (72h14m33.863983858s) [noise=true , address=145.79.227.28:4727 , piece_id=BSP57B4M3L7UIEDLVM2ASNQRK5BYLGTMAJ633QN4WFNS3OCNWDGQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.993301476 (72h14m33.864384521s) [noise=true , address=145.79.227.41:4085 , piece_id=ODJABK4R6SHNXNNQ47WI7W4YB4NR5A6QDTUE4WXVUFBVE7X3AJCQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.993926856 (72h14m33.865009922s) [noise=true , address=145.79.227.25:4619 , piece_id=CJJ7CONWO42FFSV7QMB4JSATW7BUDIS6KJXYX2SIBQMWE3UTYA4A, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.994260803 (72h14m33.865343838s) [noise=true , address=77.90.31.14:4661 , piece_id=BON7N46ZON2BYBD4UHNHPAECUF2OL7KV6B7Z2SZWVS6OGTEAK3RA, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.994756687 (72h14m33.865839732s) [noise=true , address=145.79.227.31:4799 , piece_id=VKXVHWX7LFKBH4E32F6CMAHMQX2P5AREIEIWHTXK7U22TEWDMRFQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.995123304 (72h14m33.866206349s) [noise=true , address=145.79.227.26:4653 , piece_id=LMBLPSOTMKPFS4KGAJOSRR2EM2MVKJIXSRE5MIB4SF552S2O3QYQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:35.816987512 (72h14m35.688070547s) [upload=63248]: Next returned nil segment Dec 8 15:43:35.817000286 (72h14m35.688083331s) [upload=63248]: Waiting for error group managing segments... Dec 8 15:43:36.138037836 (72h14m36.009120871s) [noise=true , address=77.90.31.10:4519 , piece_id=PGLC7MAH7IJ7GP6DTN65WUESDI3Y2JNDRVFR3QJJBNZV2H2SFJSA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.161877783 (72h14m36.032960818s) [noise=true , address=145.79.227.38:4979 , piece_id=4A4TZ2YT5KBQVYX7XSKIUMFJBLR3R7NJOZCVXEVVIK67VC744VXA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.164129480 (72h14m36.035212525s) [noise=true , address=145.79.227.31:4799 , piece_id=VKXVHWX7LFKBH4E32F6CMAHMQX2P5AREIEIWHTXK7U22TEWDMRFQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.166039237 (72h14m36.037122272s) [noise=true , address=145.79.227.41:4085 , piece_id=ODJABK4R6SHNXNNQ47WI7W4YB4NR5A6QDTUE4WXVUFBVE7X3AJCQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.166480136 (72h14m36.037563181s) [noise=true , address=145.79.227.25:4619 , piece_id=CJJ7CONWO42FFSV7QMB4JSATW7BUDIS6KJXYX2SIBQMWE3UTYA4A, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.167642099 (72h14m36.038725134s) [noise=true , address=145.79.227.43:4157 , piece_id=3P7NOOCSNAUBKH4SVL2SUSQOADSA3IKRHTPRL4P7VGH62DL23LFQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.170345487 (72h14m36.041428532s) [noise=true , address=77.90.31.19:4842 , piece_id=D6RHPHJCJTV7I6UF5HQSPFFNVVHJDYXOIY3C747PLFFACW32INWA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.173595836 (72h14m36.044678881s) [noise=true , address=145.79.227.21:4475 , piece_id=PLY2YAALNNQKXXF5MZUUKDUV4HP7XUDRD3QXJTTKQI6FCGLHEA7A, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.185245680 (72h14m36.056328725s) [noise=true , address=145.79.227.28:4727 , piece_id=BSP57B4M3L7UIEDLVM2ASNQRK5BYLGTMAJ633QN4WFNS3OCNWDGQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.188436325 (72h14m36.05951937s) [noise=true , address=77.90.31.14:4661 , piece_id=BON7N46ZON2BYBD4UHNHPAECUF2OL7KV6B7Z2SZWVS6OGTEAK3RA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.199146960 (72h14m36.070229995s) [noise=true , address=145.79.227.38:4978 , piece_id=2AI2DCS6TNPJGURTBMNDKT5N3GISWDVOC5VTLIP6MVUAEWP2V3ZQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.484809459 (72h14m36.355892504s) [noise=true , address=145.79.227.39:4015 , piece_id=AUU7NTJQWR52XFWHVONDFHZHKXPOUS2W4UL4OF4XGTWV4ABETSHQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:37.179185330 (72h14m37.050268365s) [noise=true , address=145.79.227.22:4510 , piece_id=BVTVWVC7VRYL7TREJ63BRWY6LAHU2N5AHYJR3JRHQPPEEITTMFVQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:37.179203285 (72h14m37.05028633s) [seg_pos={0 0}, upload=63248]: Segment reached optimal threshold of 13 pieces. Dec 8 15:43:37.179369642 (72h14m37.050452687s) [noise=true , address=145.79.227.26:4653 , piece_id=LMBLPSOTMKPFS4KGAJOSRR2EM2MVKJIXSRE5MIB4SF552S2O3QYQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: expected piece hash; context canceled; EOF Dec 8 15:43:37.179375573 (72h14m37.050458608s) [noise=true , address=145.79.227.26:4653 , piece_id=LMBLPSOTMKPFS4KGAJOSRR2EM2MVKJIXSRE5MIB4SF552S2O3QYQ, seg_pos={0 0}, upload=63248]: Long tail cancellation due to optimal threshold Dec 8 15:43:37.179483609 (72h14m37.050566654s) [noise=true , address=145.79.227.29:4943 , piece_id=TJFW4JKEHHCDTMPVS46BI4OYF35BVHE3VIFM6LD2CJKRFCZXL5FA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 8 15:43:37.179488698 (72h14m37.050571743s) [noise=true , address=145.79.227.29:4943 , piece_id=TJFW4JKEHHCDTMPVS46BI4OYF35BVHE3VIFM6LD2CJKRFCZXL5FA, seg_pos={0 0}, upload=63248]: Long tail cancellation due to optimal threshold Dec 8 15:43:37.179679632 (72h14m37.050762687s) [noise=true , address=145.79.227.27:4689 , piece_id=LAMO54F6RV2YAJQNHCIOJNSAQGCXK23WHVJJ6VC2REYJ7LUW5R3Q, seg_pos={0 0}, upload=63248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ): protocol: write tcp 188.166.225.170:57686->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:57686->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57686->145.79.227.27:4689: use of closed network connection Dec 8 15:43:37.179722844 (72h14m37.050805879s) [noise=true , address=145.79.227.27:4689 , piece_id=LAMO54F6RV2YAJQNHCIOJNSAQGCXK23WHVJJ6VC2REYJ7LUW5R3Q, seg_pos={0 0}, upload=63248]: Long tail cancellation due to optimal threshold Dec 8 15:43:37.179754174 (72h14m37.050837209s) [seg_pos={0 0}, upload=63248]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 15:43:37.179977008 (72h14m37.051060043s) [upload=63248]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 15:43:37.179985915 (72h14m37.05106895s) [upload=63248]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 15:43:37.492140220 (72h14m37.363223265s) [upload=63248]: Done uploading segments... Dec 8 15:43:37.492158695 (72h14m37.36324173s) [upload=63248]: Upload finished. err: <nil> Duration: 7.224710665s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [X.X.XXXXX.XXX..X....X...X...X.X.X.....XX.X.X.XXXX.X...XX.X..] duration: 3.579849616s trace: https://jaeger.dev.storj.io/trace/4128707c24eac82c FAIL (2025-12-08 15:47:45 UTC) orbiter warning: context deadline exceeded Duration: 3.579955537s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.898619431s trace: https://jaeger.dev.storj.io/trace/4195524620a3e2c2 FAIL (2025-12-08 15:47:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments... Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces. Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments... Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.898646734s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................X.........X...X..............X......] duration: 8.52609357s trace: https://jaeger.dev.storj.io/trace/3711a2689ed652fd FAIL (2025-12-08 15:47:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: VKXVHWX7LFKBH4E32F6CMAHMQX2P5AREIEIWHTXK7U22TEWDMRFQ) context deadline exceeded; piecestore: (Node ID: 18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, Piece ID: AUU7NTJQWR52XFWHVONDFHZHKXPOUS2W4UL4OF4XGTWV4ABETSHQ) context deadline exceeded; piecestore: (Node ID: 12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, Piece ID: 2AI2DCS6TNPJGURTBMNDKT5N3GISWDVOC5VTLIP6MVUAEWP2V3ZQ) context deadline exceeded ======LOGS====== Dec 8 15:43:33.737263649 (72h14m33.608346694s) [upload=63248]: Starting upload Dec 8 15:43:33.737410899 (72h14m33.608493944s) [upload=63248]: Done starting upload Dec 8 15:43:33.737492094 (72h14m33.608575149s) [upload=63248]: Uploading segments... Dec 8 15:43:33.739621138 (72h14m33.610704183s) [upload=63248]: Got next segment. Inline: false Dec 8 15:43:33.739675270 (72h14m33.610758305s) [upload=63248]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 15:43:33.739681903 (72h14m33.610764938s) [upload=63248]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 15:43:33.988217234 (72h14m33.859300269s) [seg_pos={0 0}, upload=63248]: Begin upload segment... Dec 8 15:43:33.988366548 (72h14m33.859449583s) [seg_pos={0 0}, upload=63248]: Done begin upload segment. Dec 8 15:43:33.988395153 (72h14m33.859478198s) [noise=true , address=77.90.31.19:4842 , piece_id=D6RHPHJCJTV7I6UF5HQSPFFNVVHJDYXOIY3C747PLFFACW32INWA, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.988434698 (72h14m33.859517763s) [noise=true , address=145.79.227.27:4689 , piece_id=LAMO54F6RV2YAJQNHCIOJNSAQGCXK23WHVJJ6VC2REYJ7LUW5R3Q, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.989099694 (72h14m33.860182729s) [noise=true , address=145.79.227.29:4943 , piece_id=TJFW4JKEHHCDTMPVS46BI4OYF35BVHE3VIFM6LD2CJKRFCZXL5FA, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.989662838 (72h14m33.860745873s) [noise=true , address=145.79.227.22:4510 , piece_id=BVTVWVC7VRYL7TREJ63BRWY6LAHU2N5AHYJR3JRHQPPEEITTMFVQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.990143793 (72h14m33.861226828s) [noise=true , address=145.79.227.43:4157 , piece_id=3P7NOOCSNAUBKH4SVL2SUSQOADSA3IKRHTPRL4P7VGH62DL23LFQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.990589972 (72h14m33.861673007s) [noise=true , address=77.90.31.10:4519 , piece_id=PGLC7MAH7IJ7GP6DTN65WUESDI3Y2JNDRVFR3QJJBNZV2H2SFJSA, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.991038847 (72h14m33.862121882s) [noise=true , address=145.79.227.39:4015 , piece_id=AUU7NTJQWR52XFWHVONDFHZHKXPOUS2W4UL4OF4XGTWV4ABETSHQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.991412218 (72h14m33.862495263s) [noise=true , address=145.79.227.38:4978 , piece_id=2AI2DCS6TNPJGURTBMNDKT5N3GISWDVOC5VTLIP6MVUAEWP2V3ZQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.991728711 (72h14m33.862811746s) [noise=true , address=145.79.227.38:4979 , piece_id=4A4TZ2YT5KBQVYX7XSKIUMFJBLR3R7NJOZCVXEVVIK67VC744VXA, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.992474100 (72h14m33.863557145s) [noise=true , address=145.79.227.21:4475 , piece_id=PLY2YAALNNQKXXF5MZUUKDUV4HP7XUDRD3QXJTTKQI6FCGLHEA7A, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.992900823 (72h14m33.863983858s) [noise=true , address=145.79.227.28:4727 , piece_id=BSP57B4M3L7UIEDLVM2ASNQRK5BYLGTMAJ633QN4WFNS3OCNWDGQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.993301476 (72h14m33.864384521s) [noise=true , address=145.79.227.41:4085 , piece_id=ODJABK4R6SHNXNNQ47WI7W4YB4NR5A6QDTUE4WXVUFBVE7X3AJCQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.993926856 (72h14m33.865009922s) [noise=true , address=145.79.227.25:4619 , piece_id=CJJ7CONWO42FFSV7QMB4JSATW7BUDIS6KJXYX2SIBQMWE3UTYA4A, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.994260803 (72h14m33.865343838s) [noise=true , address=77.90.31.14:4661 , piece_id=BON7N46ZON2BYBD4UHNHPAECUF2OL7KV6B7Z2SZWVS6OGTEAK3RA, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.994756687 (72h14m33.865839732s) [noise=true , address=145.79.227.31:4799 , piece_id=VKXVHWX7LFKBH4E32F6CMAHMQX2P5AREIEIWHTXK7U22TEWDMRFQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.995123304 (72h14m33.866206349s) [noise=true , address=145.79.227.26:4653 , piece_id=LMBLPSOTMKPFS4KGAJOSRR2EM2MVKJIXSRE5MIB4SF552S2O3QYQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:35.816987512 (72h14m35.688070547s) [upload=63248]: Next returned nil segment Dec 8 15:43:35.817000286 (72h14m35.688083331s) [upload=63248]: Waiting for error group managing segments... Dec 8 15:43:36.138037836 (72h14m36.009120871s) [noise=true , address=77.90.31.10:4519 , piece_id=PGLC7MAH7IJ7GP6DTN65WUESDI3Y2JNDRVFR3QJJBNZV2H2SFJSA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.161877783 (72h14m36.032960818s) [noise=true , address=145.79.227.38:4979 , piece_id=4A4TZ2YT5KBQVYX7XSKIUMFJBLR3R7NJOZCVXEVVIK67VC744VXA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.164129480 (72h14m36.035212525s) [noise=true , address=145.79.227.31:4799 , piece_id=VKXVHWX7LFKBH4E32F6CMAHMQX2P5AREIEIWHTXK7U22TEWDMRFQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.166039237 (72h14m36.037122272s) [noise=true , address=145.79.227.41:4085 , piece_id=ODJABK4R6SHNXNNQ47WI7W4YB4NR5A6QDTUE4WXVUFBVE7X3AJCQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.166480136 (72h14m36.037563181s) [noise=true , address=145.79.227.25:4619 , piece_id=CJJ7CONWO42FFSV7QMB4JSATW7BUDIS6KJXYX2SIBQMWE3UTYA4A, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.167642099 (72h14m36.038725134s) [noise=true , address=145.79.227.43:4157 , piece_id=3P7NOOCSNAUBKH4SVL2SUSQOADSA3IKRHTPRL4P7VGH62DL23LFQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.170345487 (72h14m36.041428532s) [noise=true , address=77.90.31.19:4842 , piece_id=D6RHPHJCJTV7I6UF5HQSPFFNVVHJDYXOIY3C747PLFFACW32INWA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.173595836 (72h14m36.044678881s) [noise=true , address=145.79.227.21:4475 , piece_id=PLY2YAALNNQKXXF5MZUUKDUV4HP7XUDRD3QXJTTKQI6FCGLHEA7A, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.185245680 (72h14m36.056328725s) [noise=true , address=145.79.227.28:4727 , piece_id=BSP57B4M3L7UIEDLVM2ASNQRK5BYLGTMAJ633QN4WFNS3OCNWDGQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.188436325 (72h14m36.05951937s) [noise=true , address=77.90.31.14:4661 , piece_id=BON7N46ZON2BYBD4UHNHPAECUF2OL7KV6B7Z2SZWVS6OGTEAK3RA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.199146960 (72h14m36.070229995s) [noise=true , address=145.79.227.38:4978 , piece_id=2AI2DCS6TNPJGURTBMNDKT5N3GISWDVOC5VTLIP6MVUAEWP2V3ZQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.484809459 (72h14m36.355892504s) [noise=true , address=145.79.227.39:4015 , piece_id=AUU7NTJQWR52XFWHVONDFHZHKXPOUS2W4UL4OF4XGTWV4ABETSHQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:37.179185330 (72h14m37.050268365s) [noise=true , address=145.79.227.22:4510 , piece_id=BVTVWVC7VRYL7TREJ63BRWY6LAHU2N5AHYJR3JRHQPPEEITTMFVQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:37.179203285 (72h14m37.05028633s) [seg_pos={0 0}, upload=63248]: Segment reached optimal threshold of 13 pieces. Dec 8 15:43:37.179369642 (72h14m37.050452687s) [noise=true , address=145.79.227.26:4653 , piece_id=LMBLPSOTMKPFS4KGAJOSRR2EM2MVKJIXSRE5MIB4SF552S2O3QYQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: expected piece hash; context canceled; EOF Dec 8 15:43:37.179375573 (72h14m37.050458608s) [noise=true , address=145.79.227.26:4653 , piece_id=LMBLPSOTMKPFS4KGAJOSRR2EM2MVKJIXSRE5MIB4SF552S2O3QYQ, seg_pos={0 0}, upload=63248]: Long tail cancellation due to optimal threshold Dec 8 15:43:37.179483609 (72h14m37.050566654s) [noise=true , address=145.79.227.29:4943 , piece_id=TJFW4JKEHHCDTMPVS46BI4OYF35BVHE3VIFM6LD2CJKRFCZXL5FA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 8 15:43:37.179488698 (72h14m37.050571743s) [noise=true , address=145.79.227.29:4943 , piece_id=TJFW4JKEHHCDTMPVS46BI4OYF35BVHE3VIFM6LD2CJKRFCZXL5FA, seg_pos={0 0}, upload=63248]: Long tail cancellation due to optimal threshold Dec 8 15:43:37.179679632 (72h14m37.050762687s) [noise=true , address=145.79.227.27:4689 , piece_id=LAMO54F6RV2YAJQNHCIOJNSAQGCXK23WHVJJ6VC2REYJ7LUW5R3Q, seg_pos={0 0}, upload=63248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ): protocol: write tcp 188.166.225.170:57686->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:57686->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57686->145.79.227.27:4689: use of closed network connection Dec 8 15:43:37.179722844 (72h14m37.050805879s) [noise=true , address=145.79.227.27:4689 , piece_id=LAMO54F6RV2YAJQNHCIOJNSAQGCXK23WHVJJ6VC2REYJ7LUW5R3Q, seg_pos={0 0}, upload=63248]: Long tail cancellation due to optimal threshold Dec 8 15:43:37.179754174 (72h14m37.050837209s) [seg_pos={0 0}, upload=63248]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 15:43:37.179977008 (72h14m37.051060043s) [upload=63248]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 15:43:37.179985915 (72h14m37.05106895s) [upload=63248]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 15:43:37.492140220 (72h14m37.363223265s) [upload=63248]: Done uploading segments... Dec 8 15:43:37.492158695 (72h14m37.36324173s) [upload=63248]: Upload finished. err: <nil> Duration: 8.526116804s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.334557247s trace: https://jaeger.dev.storj.io/trace/153706e03a29fd88 FAIL (2025-12-08 15:47:33 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 2.334643852s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.......................X.........................X..X...X...] duration: 5.750613636s trace: https://jaeger.dev.storj.io/trace/33a3f0149193afa6 FAIL (2025-12-08 15:47:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.75064176s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [XX.X.XXXXX.XXX..X....X...X...X.X.X.....XX.X.X.XXXX.X...XX.X.] duration: 3.509486009s trace: https://jaeger.dev.storj.io/trace/4128707c24eac82c FAIL (2025-12-08 15:47:26 UTC) orbiter warning: context deadline exceeded Duration: 3.509536444s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........................X.........X...X..............X.....] duration: 6.823833275s trace: https://jaeger.dev.storj.io/trace/3711a2689ed652fd FAIL (2025-12-08 15:47:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: VKXVHWX7LFKBH4E32F6CMAHMQX2P5AREIEIWHTXK7U22TEWDMRFQ) context deadline exceeded; piecestore: (Node ID: 18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, Piece ID: AUU7NTJQWR52XFWHVONDFHZHKXPOUS2W4UL4OF4XGTWV4ABETSHQ) context deadline exceeded; piecestore: (Node ID: 12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, Piece ID: 2AI2DCS6TNPJGURTBMNDKT5N3GISWDVOC5VTLIP6MVUAEWP2V3ZQ) context deadline exceeded ======LOGS====== Dec 8 15:43:33.737263649 (72h14m33.608346694s) [upload=63248]: Starting upload Dec 8 15:43:33.737410899 (72h14m33.608493944s) [upload=63248]: Done starting upload Dec 8 15:43:33.737492094 (72h14m33.608575149s) [upload=63248]: Uploading segments... Dec 8 15:43:33.739621138 (72h14m33.610704183s) [upload=63248]: Got next segment. Inline: false Dec 8 15:43:33.739675270 (72h14m33.610758305s) [upload=63248]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 15:43:33.739681903 (72h14m33.610764938s) [upload=63248]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 15:43:33.988217234 (72h14m33.859300269s) [seg_pos={0 0}, upload=63248]: Begin upload segment... Dec 8 15:43:33.988366548 (72h14m33.859449583s) [seg_pos={0 0}, upload=63248]: Done begin upload segment. Dec 8 15:43:33.988395153 (72h14m33.859478198s) [noise=true , address=77.90.31.19:4842 , piece_id=D6RHPHJCJTV7I6UF5HQSPFFNVVHJDYXOIY3C747PLFFACW32INWA, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.988434698 (72h14m33.859517763s) [noise=true , address=145.79.227.27:4689 , piece_id=LAMO54F6RV2YAJQNHCIOJNSAQGCXK23WHVJJ6VC2REYJ7LUW5R3Q, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.989099694 (72h14m33.860182729s) [noise=true , address=145.79.227.29:4943 , piece_id=TJFW4JKEHHCDTMPVS46BI4OYF35BVHE3VIFM6LD2CJKRFCZXL5FA, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.989662838 (72h14m33.860745873s) [noise=true , address=145.79.227.22:4510 , piece_id=BVTVWVC7VRYL7TREJ63BRWY6LAHU2N5AHYJR3JRHQPPEEITTMFVQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.990143793 (72h14m33.861226828s) [noise=true , address=145.79.227.43:4157 , piece_id=3P7NOOCSNAUBKH4SVL2SUSQOADSA3IKRHTPRL4P7VGH62DL23LFQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.990589972 (72h14m33.861673007s) [noise=true , address=77.90.31.10:4519 , piece_id=PGLC7MAH7IJ7GP6DTN65WUESDI3Y2JNDRVFR3QJJBNZV2H2SFJSA, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.991038847 (72h14m33.862121882s) [noise=true , address=145.79.227.39:4015 , piece_id=AUU7NTJQWR52XFWHVONDFHZHKXPOUS2W4UL4OF4XGTWV4ABETSHQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.991412218 (72h14m33.862495263s) [noise=true , address=145.79.227.38:4978 , piece_id=2AI2DCS6TNPJGURTBMNDKT5N3GISWDVOC5VTLIP6MVUAEWP2V3ZQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.991728711 (72h14m33.862811746s) [noise=true , address=145.79.227.38:4979 , piece_id=4A4TZ2YT5KBQVYX7XSKIUMFJBLR3R7NJOZCVXEVVIK67VC744VXA, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.992474100 (72h14m33.863557145s) [noise=true , address=145.79.227.21:4475 , piece_id=PLY2YAALNNQKXXF5MZUUKDUV4HP7XUDRD3QXJTTKQI6FCGLHEA7A, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.992900823 (72h14m33.863983858s) [noise=true , address=145.79.227.28:4727 , piece_id=BSP57B4M3L7UIEDLVM2ASNQRK5BYLGTMAJ633QN4WFNS3OCNWDGQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.993301476 (72h14m33.864384521s) [noise=true , address=145.79.227.41:4085 , piece_id=ODJABK4R6SHNXNNQ47WI7W4YB4NR5A6QDTUE4WXVUFBVE7X3AJCQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.993926856 (72h14m33.865009922s) [noise=true , address=145.79.227.25:4619 , piece_id=CJJ7CONWO42FFSV7QMB4JSATW7BUDIS6KJXYX2SIBQMWE3UTYA4A, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.994260803 (72h14m33.865343838s) [noise=true , address=77.90.31.14:4661 , piece_id=BON7N46ZON2BYBD4UHNHPAECUF2OL7KV6B7Z2SZWVS6OGTEAK3RA, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.994756687 (72h14m33.865839732s) [noise=true , address=145.79.227.31:4799 , piece_id=VKXVHWX7LFKBH4E32F6CMAHMQX2P5AREIEIWHTXK7U22TEWDMRFQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.995123304 (72h14m33.866206349s) [noise=true , address=145.79.227.26:4653 , piece_id=LMBLPSOTMKPFS4KGAJOSRR2EM2MVKJIXSRE5MIB4SF552S2O3QYQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:35.816987512 (72h14m35.688070547s) [upload=63248]: Next returned nil segment Dec 8 15:43:35.817000286 (72h14m35.688083331s) [upload=63248]: Waiting for error group managing segments... Dec 8 15:43:36.138037836 (72h14m36.009120871s) [noise=true , address=77.90.31.10:4519 , piece_id=PGLC7MAH7IJ7GP6DTN65WUESDI3Y2JNDRVFR3QJJBNZV2H2SFJSA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.161877783 (72h14m36.032960818s) [noise=true , address=145.79.227.38:4979 , piece_id=4A4TZ2YT5KBQVYX7XSKIUMFJBLR3R7NJOZCVXEVVIK67VC744VXA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.164129480 (72h14m36.035212525s) [noise=true , address=145.79.227.31:4799 , piece_id=VKXVHWX7LFKBH4E32F6CMAHMQX2P5AREIEIWHTXK7U22TEWDMRFQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.166039237 (72h14m36.037122272s) [noise=true , address=145.79.227.41:4085 , piece_id=ODJABK4R6SHNXNNQ47WI7W4YB4NR5A6QDTUE4WXVUFBVE7X3AJCQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.166480136 (72h14m36.037563181s) [noise=true , address=145.79.227.25:4619 , piece_id=CJJ7CONWO42FFSV7QMB4JSATW7BUDIS6KJXYX2SIBQMWE3UTYA4A, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.167642099 (72h14m36.038725134s) [noise=true , address=145.79.227.43:4157 , piece_id=3P7NOOCSNAUBKH4SVL2SUSQOADSA3IKRHTPRL4P7VGH62DL23LFQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.170345487 (72h14m36.041428532s) [noise=true , address=77.90.31.19:4842 , piece_id=D6RHPHJCJTV7I6UF5HQSPFFNVVHJDYXOIY3C747PLFFACW32INWA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.173595836 (72h14m36.044678881s) [noise=true , address=145.79.227.21:4475 , piece_id=PLY2YAALNNQKXXF5MZUUKDUV4HP7XUDRD3QXJTTKQI6FCGLHEA7A, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.185245680 (72h14m36.056328725s) [noise=true , address=145.79.227.28:4727 , piece_id=BSP57B4M3L7UIEDLVM2ASNQRK5BYLGTMAJ633QN4WFNS3OCNWDGQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.188436325 (72h14m36.05951937s) [noise=true , address=77.90.31.14:4661 , piece_id=BON7N46ZON2BYBD4UHNHPAECUF2OL7KV6B7Z2SZWVS6OGTEAK3RA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.199146960 (72h14m36.070229995s) [noise=true , address=145.79.227.38:4978 , piece_id=2AI2DCS6TNPJGURTBMNDKT5N3GISWDVOC5VTLIP6MVUAEWP2V3ZQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.484809459 (72h14m36.355892504s) [noise=true , address=145.79.227.39:4015 , piece_id=AUU7NTJQWR52XFWHVONDFHZHKXPOUS2W4UL4OF4XGTWV4ABETSHQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:37.179185330 (72h14m37.050268365s) [noise=true , address=145.79.227.22:4510 , piece_id=BVTVWVC7VRYL7TREJ63BRWY6LAHU2N5AHYJR3JRHQPPEEITTMFVQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:37.179203285 (72h14m37.05028633s) [seg_pos={0 0}, upload=63248]: Segment reached optimal threshold of 13 pieces. Dec 8 15:43:37.179369642 (72h14m37.050452687s) [noise=true , address=145.79.227.26:4653 , piece_id=LMBLPSOTMKPFS4KGAJOSRR2EM2MVKJIXSRE5MIB4SF552S2O3QYQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: expected piece hash; context canceled; EOF Dec 8 15:43:37.179375573 (72h14m37.050458608s) [noise=true , address=145.79.227.26:4653 , piece_id=LMBLPSOTMKPFS4KGAJOSRR2EM2MVKJIXSRE5MIB4SF552S2O3QYQ, seg_pos={0 0}, upload=63248]: Long tail cancellation due to optimal threshold Dec 8 15:43:37.179483609 (72h14m37.050566654s) [noise=true , address=145.79.227.29:4943 , piece_id=TJFW4JKEHHCDTMPVS46BI4OYF35BVHE3VIFM6LD2CJKRFCZXL5FA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 8 15:43:37.179488698 (72h14m37.050571743s) [noise=true , address=145.79.227.29:4943 , piece_id=TJFW4JKEHHCDTMPVS46BI4OYF35BVHE3VIFM6LD2CJKRFCZXL5FA, seg_pos={0 0}, upload=63248]: Long tail cancellation due to optimal threshold Dec 8 15:43:37.179679632 (72h14m37.050762687s) [noise=true , address=145.79.227.27:4689 , piece_id=LAMO54F6RV2YAJQNHCIOJNSAQGCXK23WHVJJ6VC2REYJ7LUW5R3Q, seg_pos={0 0}, upload=63248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ): protocol: write tcp 188.166.225.170:57686->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:57686->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57686->145.79.227.27:4689: use of closed network connection Dec 8 15:43:37.179722844 (72h14m37.050805879s) [noise=true , address=145.79.227.27:4689 , piece_id=LAMO54F6RV2YAJQNHCIOJNSAQGCXK23WHVJJ6VC2REYJ7LUW5R3Q, seg_pos={0 0}, upload=63248]: Long tail cancellation due to optimal threshold Dec 8 15:43:37.179754174 (72h14m37.050837209s) [seg_pos={0 0}, upload=63248]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 15:43:37.179977008 (72h14m37.051060043s) [upload=63248]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 15:43:37.179985915 (72h14m37.05106895s) [upload=63248]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 15:43:37.492140220 (72h14m37.363223265s) [upload=63248]: Done uploading segments... Dec 8 15:43:37.492158695 (72h14m37.36324173s) [upload=63248]: Upload finished. err: <nil> Duration: 6.823864946s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.24455389s trace: https://jaeger.dev.storj.io/trace/54ba7b1e9d79a182 FAIL (2025-12-08 15:47:09 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.244581611s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [......X............X.X.XX.XX.XX..X.....XX...X.X.X.X....XX...] duration: 19.512298646s trace: https://jaeger.dev.storj.io/trace/34dcbf1241a033f7 FAIL (2025-12-08 15:47:05 UTC) orbiter warning: context deadline exceeded Duration: 19.512323764s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [........................X.........................X..X...X..] duration: 42.829826416s trace: https://jaeger.dev.storj.io/trace/33a3f0149193afa6 FAIL (2025-12-08 15:47:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 42.829856563s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [XXX.X.XXXXX.XXX..X....X...X...X.X.X.....XX.X.X.XXXX.X...XX.X] duration: 5.001632859s trace: https://jaeger.dev.storj.io/trace/4128707c24eac82c FAIL (2025-12-08 15:46:59 UTC) orbiter warning: context deadline exceeded Duration: 5.001656595s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.89049995s trace: https://jaeger.dev.storj.io/trace/153706e03a29fd88 FAIL (2025-12-08 15:46:38 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 5.890535578s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.360795264s trace: https://jaeger.dev.storj.io/trace/4195524620a3e2c2 FAIL (2025-12-08 15:46:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments... Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces. Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments... Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.360822555s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.......X............X.X.XX.XX.XX..X.....XX...X.X.X.X....XX..] duration: 22.206040264s trace: https://jaeger.dev.storj.io/trace/34dcbf1241a033f7 FAIL (2025-12-08 15:46:32 UTC) orbiter warning: context deadline exceeded Duration: 22.206070371s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [XXXX.X.XXXXX.XXX..X....X...X...X.X.X.....XX.X.X.XXXX.X...XX.] duration: 3.472741821s trace: https://jaeger.dev.storj.io/trace/309b74d3892091a7 FAIL (2025-12-08 15:46:32 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.472768993s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........................X.........X...X..............X....] duration: 8.514800968s trace: https://jaeger.dev.storj.io/trace/3711a2689ed652fd FAIL (2025-12-08 15:46:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: VKXVHWX7LFKBH4E32F6CMAHMQX2P5AREIEIWHTXK7U22TEWDMRFQ) context deadline exceeded; piecestore: (Node ID: 18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, Piece ID: AUU7NTJQWR52XFWHVONDFHZHKXPOUS2W4UL4OF4XGTWV4ABETSHQ) context deadline exceeded; piecestore: (Node ID: 12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, Piece ID: 2AI2DCS6TNPJGURTBMNDKT5N3GISWDVOC5VTLIP6MVUAEWP2V3ZQ) context deadline exceeded ======LOGS====== Dec 8 15:43:33.737263649 (72h14m33.608346694s) [upload=63248]: Starting upload Dec 8 15:43:33.737410899 (72h14m33.608493944s) [upload=63248]: Done starting upload Dec 8 15:43:33.737492094 (72h14m33.608575149s) [upload=63248]: Uploading segments... Dec 8 15:43:33.739621138 (72h14m33.610704183s) [upload=63248]: Got next segment. Inline: false Dec 8 15:43:33.739675270 (72h14m33.610758305s) [upload=63248]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 15:43:33.739681903 (72h14m33.610764938s) [upload=63248]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 15:43:33.988217234 (72h14m33.859300269s) [seg_pos={0 0}, upload=63248]: Begin upload segment... Dec 8 15:43:33.988366548 (72h14m33.859449583s) [seg_pos={0 0}, upload=63248]: Done begin upload segment. Dec 8 15:43:33.988395153 (72h14m33.859478198s) [noise=true , address=77.90.31.19:4842 , piece_id=D6RHPHJCJTV7I6UF5HQSPFFNVVHJDYXOIY3C747PLFFACW32INWA, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.988434698 (72h14m33.859517763s) [noise=true , address=145.79.227.27:4689 , piece_id=LAMO54F6RV2YAJQNHCIOJNSAQGCXK23WHVJJ6VC2REYJ7LUW5R3Q, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.989099694 (72h14m33.860182729s) [noise=true , address=145.79.227.29:4943 , piece_id=TJFW4JKEHHCDTMPVS46BI4OYF35BVHE3VIFM6LD2CJKRFCZXL5FA, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.989662838 (72h14m33.860745873s) [noise=true , address=145.79.227.22:4510 , piece_id=BVTVWVC7VRYL7TREJ63BRWY6LAHU2N5AHYJR3JRHQPPEEITTMFVQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.990143793 (72h14m33.861226828s) [noise=true , address=145.79.227.43:4157 , piece_id=3P7NOOCSNAUBKH4SVL2SUSQOADSA3IKRHTPRL4P7VGH62DL23LFQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.990589972 (72h14m33.861673007s) [noise=true , address=77.90.31.10:4519 , piece_id=PGLC7MAH7IJ7GP6DTN65WUESDI3Y2JNDRVFR3QJJBNZV2H2SFJSA, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.991038847 (72h14m33.862121882s) [noise=true , address=145.79.227.39:4015 , piece_id=AUU7NTJQWR52XFWHVONDFHZHKXPOUS2W4UL4OF4XGTWV4ABETSHQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.991412218 (72h14m33.862495263s) [noise=true , address=145.79.227.38:4978 , piece_id=2AI2DCS6TNPJGURTBMNDKT5N3GISWDVOC5VTLIP6MVUAEWP2V3ZQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.991728711 (72h14m33.862811746s) [noise=true , address=145.79.227.38:4979 , piece_id=4A4TZ2YT5KBQVYX7XSKIUMFJBLR3R7NJOZCVXEVVIK67VC744VXA, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.992474100 (72h14m33.863557145s) [noise=true , address=145.79.227.21:4475 , piece_id=PLY2YAALNNQKXXF5MZUUKDUV4HP7XUDRD3QXJTTKQI6FCGLHEA7A, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.992900823 (72h14m33.863983858s) [noise=true , address=145.79.227.28:4727 , piece_id=BSP57B4M3L7UIEDLVM2ASNQRK5BYLGTMAJ633QN4WFNS3OCNWDGQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.993301476 (72h14m33.864384521s) [noise=true , address=145.79.227.41:4085 , piece_id=ODJABK4R6SHNXNNQ47WI7W4YB4NR5A6QDTUE4WXVUFBVE7X3AJCQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.993926856 (72h14m33.865009922s) [noise=true , address=145.79.227.25:4619 , piece_id=CJJ7CONWO42FFSV7QMB4JSATW7BUDIS6KJXYX2SIBQMWE3UTYA4A, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.994260803 (72h14m33.865343838s) [noise=true , address=77.90.31.14:4661 , piece_id=BON7N46ZON2BYBD4UHNHPAECUF2OL7KV6B7Z2SZWVS6OGTEAK3RA, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.994756687 (72h14m33.865839732s) [noise=true , address=145.79.227.31:4799 , piece_id=VKXVHWX7LFKBH4E32F6CMAHMQX2P5AREIEIWHTXK7U22TEWDMRFQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.995123304 (72h14m33.866206349s) [noise=true , address=145.79.227.26:4653 , piece_id=LMBLPSOTMKPFS4KGAJOSRR2EM2MVKJIXSRE5MIB4SF552S2O3QYQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:35.816987512 (72h14m35.688070547s) [upload=63248]: Next returned nil segment Dec 8 15:43:35.817000286 (72h14m35.688083331s) [upload=63248]: Waiting for error group managing segments... Dec 8 15:43:36.138037836 (72h14m36.009120871s) [noise=true , address=77.90.31.10:4519 , piece_id=PGLC7MAH7IJ7GP6DTN65WUESDI3Y2JNDRVFR3QJJBNZV2H2SFJSA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.161877783 (72h14m36.032960818s) [noise=true , address=145.79.227.38:4979 , piece_id=4A4TZ2YT5KBQVYX7XSKIUMFJBLR3R7NJOZCVXEVVIK67VC744VXA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.164129480 (72h14m36.035212525s) [noise=true , address=145.79.227.31:4799 , piece_id=VKXVHWX7LFKBH4E32F6CMAHMQX2P5AREIEIWHTXK7U22TEWDMRFQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.166039237 (72h14m36.037122272s) [noise=true , address=145.79.227.41:4085 , piece_id=ODJABK4R6SHNXNNQ47WI7W4YB4NR5A6QDTUE4WXVUFBVE7X3AJCQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.166480136 (72h14m36.037563181s) [noise=true , address=145.79.227.25:4619 , piece_id=CJJ7CONWO42FFSV7QMB4JSATW7BUDIS6KJXYX2SIBQMWE3UTYA4A, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.167642099 (72h14m36.038725134s) [noise=true , address=145.79.227.43:4157 , piece_id=3P7NOOCSNAUBKH4SVL2SUSQOADSA3IKRHTPRL4P7VGH62DL23LFQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.170345487 (72h14m36.041428532s) [noise=true , address=77.90.31.19:4842 , piece_id=D6RHPHJCJTV7I6UF5HQSPFFNVVHJDYXOIY3C747PLFFACW32INWA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.173595836 (72h14m36.044678881s) [noise=true , address=145.79.227.21:4475 , piece_id=PLY2YAALNNQKXXF5MZUUKDUV4HP7XUDRD3QXJTTKQI6FCGLHEA7A, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.185245680 (72h14m36.056328725s) [noise=true , address=145.79.227.28:4727 , piece_id=BSP57B4M3L7UIEDLVM2ASNQRK5BYLGTMAJ633QN4WFNS3OCNWDGQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.188436325 (72h14m36.05951937s) [noise=true , address=77.90.31.14:4661 , piece_id=BON7N46ZON2BYBD4UHNHPAECUF2OL7KV6B7Z2SZWVS6OGTEAK3RA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.199146960 (72h14m36.070229995s) [noise=true , address=145.79.227.38:4978 , piece_id=2AI2DCS6TNPJGURTBMNDKT5N3GISWDVOC5VTLIP6MVUAEWP2V3ZQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.484809459 (72h14m36.355892504s) [noise=true , address=145.79.227.39:4015 , piece_id=AUU7NTJQWR52XFWHVONDFHZHKXPOUS2W4UL4OF4XGTWV4ABETSHQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:37.179185330 (72h14m37.050268365s) [noise=true , address=145.79.227.22:4510 , piece_id=BVTVWVC7VRYL7TREJ63BRWY6LAHU2N5AHYJR3JRHQPPEEITTMFVQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:37.179203285 (72h14m37.05028633s) [seg_pos={0 0}, upload=63248]: Segment reached optimal threshold of 13 pieces. Dec 8 15:43:37.179369642 (72h14m37.050452687s) [noise=true , address=145.79.227.26:4653 , piece_id=LMBLPSOTMKPFS4KGAJOSRR2EM2MVKJIXSRE5MIB4SF552S2O3QYQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: expected piece hash; context canceled; EOF Dec 8 15:43:37.179375573 (72h14m37.050458608s) [noise=true , address=145.79.227.26:4653 , piece_id=LMBLPSOTMKPFS4KGAJOSRR2EM2MVKJIXSRE5MIB4SF552S2O3QYQ, seg_pos={0 0}, upload=63248]: Long tail cancellation due to optimal threshold Dec 8 15:43:37.179483609 (72h14m37.050566654s) [noise=true , address=145.79.227.29:4943 , piece_id=TJFW4JKEHHCDTMPVS46BI4OYF35BVHE3VIFM6LD2CJKRFCZXL5FA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 8 15:43:37.179488698 (72h14m37.050571743s) [noise=true , address=145.79.227.29:4943 , piece_id=TJFW4JKEHHCDTMPVS46BI4OYF35BVHE3VIFM6LD2CJKRFCZXL5FA, seg_pos={0 0}, upload=63248]: Long tail cancellation due to optimal threshold Dec 8 15:43:37.179679632 (72h14m37.050762687s) [noise=true , address=145.79.227.27:4689 , piece_id=LAMO54F6RV2YAJQNHCIOJNSAQGCXK23WHVJJ6VC2REYJ7LUW5R3Q, seg_pos={0 0}, upload=63248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ): protocol: write tcp 188.166.225.170:57686->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:57686->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57686->145.79.227.27:4689: use of closed network connection Dec 8 15:43:37.179722844 (72h14m37.050805879s) [noise=true , address=145.79.227.27:4689 , piece_id=LAMO54F6RV2YAJQNHCIOJNSAQGCXK23WHVJJ6VC2REYJ7LUW5R3Q, seg_pos={0 0}, upload=63248]: Long tail cancellation due to optimal threshold Dec 8 15:43:37.179754174 (72h14m37.050837209s) [seg_pos={0 0}, upload=63248]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 15:43:37.179977008 (72h14m37.051060043s) [upload=63248]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 15:43:37.179985915 (72h14m37.05106895s) [upload=63248]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 15:43:37.492140220 (72h14m37.363223265s) [upload=63248]: Done uploading segments... Dec 8 15:43:37.492158695 (72h14m37.36324173s) [upload=63248]: Upload finished. err: <nil> Duration: 8.514830073s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 8.828810123s trace: https://jaeger.dev.storj.io/trace/54ba7b1e9d79a182 FAIL (2025-12-08 15:46:10 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 8.828836412s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [........X............X.X.XX.XX.XX..X.....XX...X.X.X.X....XX.] duration: 27.393953911s trace: https://jaeger.dev.storj.io/trace/34dcbf1241a033f7 FAIL (2025-12-08 15:46:04 UTC) orbiter warning: context deadline exceeded Duration: 27.393985291s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.........................X.........................X..X...X.] duration: 5.827164664s trace: https://jaeger.dev.storj.io/trace/33a3f0149193afa6 FAIL (2025-12-08 15:46:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.827191867s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..........................X.........................X..X...X] duration: 1m0.001010699s trace: https://jaeger.dev.storj.io/trace/33a3f0149193afa6 FAIL (2025-12-08 15:45:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001030717s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 7.276970708s trace: https://jaeger.dev.storj.io/trace/54ba7b1e9d79a182 FAIL (2025-12-08 15:45:39 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 7.277003401s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.015789955s trace: https://jaeger.dev.storj.io/trace/153706e03a29fd88 FAIL (2025-12-08 15:45:31 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 6.015820533s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [XXXXX.X.XXXXX.XXX..X....X...X...X.X.X.....XX.X.X.XXXX.X...XX] duration: 5.001808921s trace: https://jaeger.dev.storj.io/trace/309b74d3892091a7 FAIL (2025-12-08 15:45:30 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001836754s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.297430073s trace: https://jaeger.dev.storj.io/trace/4195524620a3e2c2 FAIL (2025-12-08 15:45:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments... Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces. Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments... Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.297458347s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.357648714s trace: https://jaeger.dev.storj.io/trace/54ba7b1e9d79a182 FAIL (2025-12-08 15:45:19 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.357679883s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........................X.........X...X..............X...] duration: 6.714124665s trace: https://jaeger.dev.storj.io/trace/3711a2689ed652fd FAIL (2025-12-08 15:45:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: VKXVHWX7LFKBH4E32F6CMAHMQX2P5AREIEIWHTXK7U22TEWDMRFQ) context deadline exceeded; piecestore: (Node ID: 18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, Piece ID: AUU7NTJQWR52XFWHVONDFHZHKXPOUS2W4UL4OF4XGTWV4ABETSHQ) context deadline exceeded; piecestore: (Node ID: 12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, Piece ID: 2AI2DCS6TNPJGURTBMNDKT5N3GISWDVOC5VTLIP6MVUAEWP2V3ZQ) context deadline exceeded ======LOGS====== Dec 8 15:43:33.737263649 (72h14m33.608346694s) [upload=63248]: Starting upload Dec 8 15:43:33.737410899 (72h14m33.608493944s) [upload=63248]: Done starting upload Dec 8 15:43:33.737492094 (72h14m33.608575149s) [upload=63248]: Uploading segments... Dec 8 15:43:33.739621138 (72h14m33.610704183s) [upload=63248]: Got next segment. Inline: false Dec 8 15:43:33.739675270 (72h14m33.610758305s) [upload=63248]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 15:43:33.739681903 (72h14m33.610764938s) [upload=63248]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 15:43:33.988217234 (72h14m33.859300269s) [seg_pos={0 0}, upload=63248]: Begin upload segment... Dec 8 15:43:33.988366548 (72h14m33.859449583s) [seg_pos={0 0}, upload=63248]: Done begin upload segment. Dec 8 15:43:33.988395153 (72h14m33.859478198s) [noise=true , address=77.90.31.19:4842 , piece_id=D6RHPHJCJTV7I6UF5HQSPFFNVVHJDYXOIY3C747PLFFACW32INWA, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.988434698 (72h14m33.859517763s) [noise=true , address=145.79.227.27:4689 , piece_id=LAMO54F6RV2YAJQNHCIOJNSAQGCXK23WHVJJ6VC2REYJ7LUW5R3Q, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.989099694 (72h14m33.860182729s) [noise=true , address=145.79.227.29:4943 , piece_id=TJFW4JKEHHCDTMPVS46BI4OYF35BVHE3VIFM6LD2CJKRFCZXL5FA, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.989662838 (72h14m33.860745873s) [noise=true , address=145.79.227.22:4510 , piece_id=BVTVWVC7VRYL7TREJ63BRWY6LAHU2N5AHYJR3JRHQPPEEITTMFVQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.990143793 (72h14m33.861226828s) [noise=true , address=145.79.227.43:4157 , piece_id=3P7NOOCSNAUBKH4SVL2SUSQOADSA3IKRHTPRL4P7VGH62DL23LFQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.990589972 (72h14m33.861673007s) [noise=true , address=77.90.31.10:4519 , piece_id=PGLC7MAH7IJ7GP6DTN65WUESDI3Y2JNDRVFR3QJJBNZV2H2SFJSA, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.991038847 (72h14m33.862121882s) [noise=true , address=145.79.227.39:4015 , piece_id=AUU7NTJQWR52XFWHVONDFHZHKXPOUS2W4UL4OF4XGTWV4ABETSHQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.991412218 (72h14m33.862495263s) [noise=true , address=145.79.227.38:4978 , piece_id=2AI2DCS6TNPJGURTBMNDKT5N3GISWDVOC5VTLIP6MVUAEWP2V3ZQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.991728711 (72h14m33.862811746s) [noise=true , address=145.79.227.38:4979 , piece_id=4A4TZ2YT5KBQVYX7XSKIUMFJBLR3R7NJOZCVXEVVIK67VC744VXA, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.992474100 (72h14m33.863557145s) [noise=true , address=145.79.227.21:4475 , piece_id=PLY2YAALNNQKXXF5MZUUKDUV4HP7XUDRD3QXJTTKQI6FCGLHEA7A, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.992900823 (72h14m33.863983858s) [noise=true , address=145.79.227.28:4727 , piece_id=BSP57B4M3L7UIEDLVM2ASNQRK5BYLGTMAJ633QN4WFNS3OCNWDGQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.993301476 (72h14m33.864384521s) [noise=true , address=145.79.227.41:4085 , piece_id=ODJABK4R6SHNXNNQ47WI7W4YB4NR5A6QDTUE4WXVUFBVE7X3AJCQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.993926856 (72h14m33.865009922s) [noise=true , address=145.79.227.25:4619 , piece_id=CJJ7CONWO42FFSV7QMB4JSATW7BUDIS6KJXYX2SIBQMWE3UTYA4A, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.994260803 (72h14m33.865343838s) [noise=true , address=77.90.31.14:4661 , piece_id=BON7N46ZON2BYBD4UHNHPAECUF2OL7KV6B7Z2SZWVS6OGTEAK3RA, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.994756687 (72h14m33.865839732s) [noise=true , address=145.79.227.31:4799 , piece_id=VKXVHWX7LFKBH4E32F6CMAHMQX2P5AREIEIWHTXK7U22TEWDMRFQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:33.995123304 (72h14m33.866206349s) [noise=true , address=145.79.227.26:4653 , piece_id=LMBLPSOTMKPFS4KGAJOSRR2EM2MVKJIXSRE5MIB4SF552S2O3QYQ, seg_pos={0 0}, upload=63248]: Uploading piece... Dec 8 15:43:35.816987512 (72h14m35.688070547s) [upload=63248]: Next returned nil segment Dec 8 15:43:35.817000286 (72h14m35.688083331s) [upload=63248]: Waiting for error group managing segments... Dec 8 15:43:36.138037836 (72h14m36.009120871s) [noise=true , address=77.90.31.10:4519 , piece_id=PGLC7MAH7IJ7GP6DTN65WUESDI3Y2JNDRVFR3QJJBNZV2H2SFJSA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.161877783 (72h14m36.032960818s) [noise=true , address=145.79.227.38:4979 , piece_id=4A4TZ2YT5KBQVYX7XSKIUMFJBLR3R7NJOZCVXEVVIK67VC744VXA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.164129480 (72h14m36.035212525s) [noise=true , address=145.79.227.31:4799 , piece_id=VKXVHWX7LFKBH4E32F6CMAHMQX2P5AREIEIWHTXK7U22TEWDMRFQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.166039237 (72h14m36.037122272s) [noise=true , address=145.79.227.41:4085 , piece_id=ODJABK4R6SHNXNNQ47WI7W4YB4NR5A6QDTUE4WXVUFBVE7X3AJCQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.166480136 (72h14m36.037563181s) [noise=true , address=145.79.227.25:4619 , piece_id=CJJ7CONWO42FFSV7QMB4JSATW7BUDIS6KJXYX2SIBQMWE3UTYA4A, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.167642099 (72h14m36.038725134s) [noise=true , address=145.79.227.43:4157 , piece_id=3P7NOOCSNAUBKH4SVL2SUSQOADSA3IKRHTPRL4P7VGH62DL23LFQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.170345487 (72h14m36.041428532s) [noise=true , address=77.90.31.19:4842 , piece_id=D6RHPHJCJTV7I6UF5HQSPFFNVVHJDYXOIY3C747PLFFACW32INWA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.173595836 (72h14m36.044678881s) [noise=true , address=145.79.227.21:4475 , piece_id=PLY2YAALNNQKXXF5MZUUKDUV4HP7XUDRD3QXJTTKQI6FCGLHEA7A, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.185245680 (72h14m36.056328725s) [noise=true , address=145.79.227.28:4727 , piece_id=BSP57B4M3L7UIEDLVM2ASNQRK5BYLGTMAJ633QN4WFNS3OCNWDGQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.188436325 (72h14m36.05951937s) [noise=true , address=77.90.31.14:4661 , piece_id=BON7N46ZON2BYBD4UHNHPAECUF2OL7KV6B7Z2SZWVS6OGTEAK3RA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.199146960 (72h14m36.070229995s) [noise=true , address=145.79.227.38:4978 , piece_id=2AI2DCS6TNPJGURTBMNDKT5N3GISWDVOC5VTLIP6MVUAEWP2V3ZQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:36.484809459 (72h14m36.355892504s) [noise=true , address=145.79.227.39:4015 , piece_id=AUU7NTJQWR52XFWHVONDFHZHKXPOUS2W4UL4OF4XGTWV4ABETSHQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:37.179185330 (72h14m37.050268365s) [noise=true , address=145.79.227.22:4510 , piece_id=BVTVWVC7VRYL7TREJ63BRWY6LAHU2N5AHYJR3JRHQPPEEITTMFVQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil> Dec 8 15:43:37.179203285 (72h14m37.05028633s) [seg_pos={0 0}, upload=63248]: Segment reached optimal threshold of 13 pieces. Dec 8 15:43:37.179369642 (72h14m37.050452687s) [noise=true , address=145.79.227.26:4653 , piece_id=LMBLPSOTMKPFS4KGAJOSRR2EM2MVKJIXSRE5MIB4SF552S2O3QYQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: expected piece hash; context canceled; EOF Dec 8 15:43:37.179375573 (72h14m37.050458608s) [noise=true , address=145.79.227.26:4653 , piece_id=LMBLPSOTMKPFS4KGAJOSRR2EM2MVKJIXSRE5MIB4SF552S2O3QYQ, seg_pos={0 0}, upload=63248]: Long tail cancellation due to optimal threshold Dec 8 15:43:37.179483609 (72h14m37.050566654s) [noise=true , address=145.79.227.29:4943 , piece_id=TJFW4JKEHHCDTMPVS46BI4OYF35BVHE3VIFM6LD2CJKRFCZXL5FA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 8 15:43:37.179488698 (72h14m37.050571743s) [noise=true , address=145.79.227.29:4943 , piece_id=TJFW4JKEHHCDTMPVS46BI4OYF35BVHE3VIFM6LD2CJKRFCZXL5FA, seg_pos={0 0}, upload=63248]: Long tail cancellation due to optimal threshold Dec 8 15:43:37.179679632 (72h14m37.050762687s) [noise=true , address=145.79.227.27:4689 , piece_id=LAMO54F6RV2YAJQNHCIOJNSAQGCXK23WHVJJ6VC2REYJ7LUW5R3Q, seg_pos={0 0}, upload=63248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ): protocol: write tcp 188.166.225.170:57686->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:57686->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57686->145.79.227.27:4689: use of closed network connection Dec 8 15:43:37.179722844 (72h14m37.050805879s) [noise=true , address=145.79.227.27:4689 , piece_id=LAMO54F6RV2YAJQNHCIOJNSAQGCXK23WHVJJ6VC2REYJ7LUW5R3Q, seg_pos={0 0}, upload=63248]: Long tail cancellation due to optimal threshold Dec 8 15:43:37.179754174 (72h14m37.050837209s) [seg_pos={0 0}, upload=63248]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 15:43:37.179977008 (72h14m37.051060043s) [upload=63248]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 15:43:37.179985915 (72h14m37.05106895s) [upload=63248]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 15:43:37.492140220 (72h14m37.363223265s) [upload=63248]: Done uploading segments... Dec 8 15:43:37.492158695 (72h14m37.36324173s) [upload=63248]: Upload finished. err: <nil> Duration: 6.714155504s core uplink us-select-1

1
...
11705
11706
11707
11708
11709
...
17284